semvec/pypi-packages huggingface dataset:
vdk-lineage 0.3.1431637373
VDK Lineage plugin collects lineage (input -> job -> output) information and send it to a pre-configured destination.

# VDK Lineage

<a href="https://pypistats.org/packages/vdk-lineage" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-lineage.svg" alt="monthly download count for vdk-lineage"></a>

VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured
destination. The lineage data is send using [OpenLineage standard](https://openlineage.io)

![](vdk-lineage.png)

At POC level currently.

Currently, lineage data is collected
 - For each data job run/execution both start and end events including the status of the job (failed/succeeded)
 - For each execute query we collect input and output tables.

TODOs:
 - Collect status of the SQL query (failed, succeeded)
 - Create parent /child relationship between sql event and job run event to track them better (single job can have multiple queries)
 - Non-SQL lineage (ingest, load data,etc)
 - Extend support for all queries
 - provide more information using facets – op id, job version,
 - figure out how to visualize parent/child relationships in Marquez
 - Explore openlineage.sqlparser instead of sqllineage library as alternative


## Usage

```
pip install vdk-lineage
```

And it will start collecting lineage from job and sql queries.

To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:
```
export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stop
```

## Build and testing

In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it
vdk-greenplum 0.0.1431637373
Versatile Data Kit SDK plugin provides support for Greenplum database and greenplum transformation templates.

<a href="https://pypistats.org/packages/vdk-greenplum" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-greenplum.svg" alt="monthly download count for vdk-greenplum"></a>

This plugin allows vdk-core to interface with and execute queries against a Greenplum database.

# Usage

Run
```bash
pip install vdk-greenplum
```

After this, data jobs will have access to a Greenplum database connection, managed by Versatile Data Kit SDK.

If it is the only database plugin installed , vdk would automatically use it.
Otherwise, users need to set VDK_DB_DEFAULT_TYPE=GREENPLUM as an environment variable or set 'db_default_type' option in the data job config file (config.ini).

For example

```python
    def run(job_input: IJobInput):
        job_input.execute_query("select 'Hi Greenplum!'")
```

## Ingestion

This plugin allows users to [ingest](https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/api/job_input.py#L90) data to a Greenplum database,
which can be preferable to inserting data manually as it automatically handles serializing, packaging and sending of the data asynchronously with configurable batching and throughput.
To do so, you must set the expected variables to connect to Greenplum, plus the following environment variable:
```sh
export VDK_INGEST_METHOD_DEFAULT=GREENPLUM
```

Then, from inside the run function in a Python step, you can use the `send_object_for_ingestion` or `send_tabular_data_for_ingestion` methods to ingest your data.

# Configuration

Run vdk config-help - search for those prefixed with "GREENPLUM_" to see what configuration options are available.

# Testing

Testing this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-greenplum/requirements.txt

Run
```bash
pip install -r requirements.txt
```
vdk-huggingface 0.1.1431637373
Integrate VDK with Huggingface as both data source and target

# Huggingface

<a href="https://pypistats.org/packages/vdk-huggingface" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-huggingface.svg" alt="monthly download count for vdk-huggingface"></a>

Versatile Data Kit (VDK) plugin for integrating with Huggingface as both a data source and a target.
This plugin allows you to ingest data payloads into a Huggingface repository and makes it easier to work with datasets stored in Huggingface.


## Usage

```
pip install vdk-huggingface
```

The functionality adds new ingestion method "huggingface" which can be used like that:

```python
job_input.send_object_for_ingestion(data, method="huggingface")
```


### Configuration

(`vdk config-help` is useful command to browse all config options of your installation of vdk)

| Name                | Description                                                                 | (example)  Value        |
|---------------------|-----------------------------------------------------------------------------|-------------------------|
| HUGGINGFACE_TOKEN   | HuggingFace API token for authentication. Get one from HuggingFace Settings | ""                      |
| HUGGINGFACE_REPO_ID | HuggingFace Dataset repository ID                                           | "username/test-dataset" |



### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-ingest-http 0.2.1431637373
Versatile Data Kit SDK ingestion plugin to ingest data via http requests.

### VDK-INGEST-HTTP Plugin

<a href="https://pypistats.org/packages/vdk-ingest-http" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-ingest-http.svg" alt="monthly download count for vdk-ingest-http"></a>

This plugin provides functionality to ingest data over http.

### Usage

To use the plugin, just install it, and set the `method` attribute of `send_object_for_ingestion()`,
and `send_tabular_data_for_ingestion()` functions to _"http"_.

Example:
```python
def run(job_input: IJobInput):
    # Do something to get data for ingestion
    payload = get_some_data()

    # Ingest the data
    job_input.send_object_for_ingestion(payload=payload,
                                        destination_table="aa_test_table",
                                        method="http",
                                        target="http://example.com/data-source"
                                        )
```
The above example shows how to ingest json data. In this case, there are three arguments that are required: `payload`,
`method` and `target`. This would be fixed in the future, so that only `payload` would be required.
<br>
The payload needs to be a json object, and should contain the `destination_table` inside, using the `@table` key. For example:
```
{
    "@table": "destination_table_name",
    "column1": "value1",
    "column2": "value2",
    "column3": "value3",
}
```
The `method` attribute needs to be provided for the time being.
<br>
The `target` attribute should specify the url endpoint, where the data will be ingested.

### Testing

To develop or test locally (from the current directory)
```bash
# install dev and test dependencies
pip install -r requirements
# install the plugin in editable mode
pip install -e .
# run the tests
pytest
```
vdk-ingest-file 0.1.1431637373
Versatile Data Kit SDK ingestion plugin to ingest data into a file.

## VDK-INGEST-FILE Plugin

<a href="https://pypistats.org/packages/vdk-ingest-file" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-ingest-file.svg" alt="monthly download count for vdk-ingest-file"></a>

This plugin provides functionality to ingest data into a file. It is intended for local testing.

### Usage

To use the plugin, just install it, and set the `method` attribute of `send_object_for_ingestion()`,
and `send_tabular_data_for_ingestion()` functions to _"file"_.

Example:
```python
def run(job_input: IJobInput):
    # Do something to get data for ingestion
    payload = get_some_data()

    # Ingest the data
    job_input.send_object_for_ingestion(payload=payload,
                                        destination_table="aa_test_table",
                                        method="file",
                                        target="name_of_file"
                                        )
```
The above example shows how to ingest json data. In this case, there is only one argument that is required: `payload`.
<br>
It needs to be a json object, and can contain the `destination_table` inside, using the `@table` key. For example:
```
{
    "@table": "destination_table_name",
    "column1": "value1",
    "column2": "value2",
    "column3": "value3",
}
```
The `target` attribute is being used to specify the name of the file, where the data will be ingested. If not specified, it is constructed,
using the model, `table.<creation-timestamp>.json`.<br>
prettymd 0.1.2
A tool that formats markdown text in Chinese.
Requires Python: >=3.5
Keywords: prettymd

prettymd
--------


A tool that formats markdown text in Chinese.


Install
-------

- 从 pypi 安装
    ```shell
    pip install prettymd
    ```


Features
--------


- 在中英文之间添加空格
    ```shell
    $ python -m prettymd "我是中文nihao呀"
    我是中文 nihao 呀
    ```

- 将英文变为代码风格。指定 style = 'code'，将英文用反引号包裹起来
    ```shell
    $ python -m prettymd "我是中文nihao呀" -s code
    我是中文 `nihao` 呀
    ```

- 从文件读取内容
    ```shell
    $ cat .\tests\testfile.md
    摘要算法就是通过摘要函数f()对任意长度的数据data计算出固定长度的摘要digest，目的是为了发现原始数据是否被人篡改过。

    $ python -m prettymd -f .\tests\testfile.md
    摘要算法就是通过摘要函数 f() 对任意长度的数据 data 计算出固定长度的摘要 digest，目的是为了发现原始数据是否被人篡改过。
    ```

- 输出到指定文件
    ```shell
    $ # -o 指定输出文件路径
    $ python -m prettymd -f .\tests\testfile.md -o testfile.md

    $ cat testfile.md
    摘要算法就是通过摘要函数 f() 对任意长度的数据 data 计算出固定长度的摘要 digest，目的是为了发现原始数据是否被人篡改过。
    ```

- 代码调用
    ```python
    >>> from prettymd import format
    >>> format('这是一个示例rst文档')
    '这是一个实例 rst 文档'

    >>> from prettymd import format
    >>> format('这是一个示例rst文档', style='code')
    '这是一个实例 `rst` 文档'

    >>> # 为标题添加索引
    >>> from prettymd import format
    >>> print(format("""
    ## h2
    ### h3
    ## h22
    #### h4
    """))

    ## 1. h2
    ### 1.1. h3
    ## 2. h22
    #### 2.0.1. h4
    ```
vdk-csv 0.1.1431637373
Versatile Data Kit SDK CSV plugin to ingest, export, or manipulate csv files.

## Versatile Data Kit CSV Plugin

<a href="https://pypistats.org/packages/vdk-csv" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-csv.svg" alt="monthly download count for vdk-csv"></a>

This plugin provides functionality to ingest and export CSV files.

### Usage

To use the plugin, just install it using

```bash
pip install vdk-csv
```

Then run help to see what you can do.
```bash
vdk ingest-csv --help
```
and
```bash
vdk export-csv --help
```

For example you can ingest CSV file into tables, or export the result of an executed query to a cvs file.

The ingestion destination depends on how vdk has been configured.
See vdk config-help - search for "ingest" to check for possible ingestion configurations.
vdk-data-source-git 0.1.1431637373
Read Git repository data source

# data-source-git

<a href="https://pypistats.org/packages/vdk-data-source-git" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-data-source-git.svg" alt="monthly download count for vdk-data-source-git"></a>

Extracts content from Git repositories along with associated file metadata.

## Usage

```
pip install vdk-data-source-git
```

### Extracted Data Schema

The extracted data is returned in a `DataSourcePayload` object with two main components: `content` and `metadata`.

#### `content`

The `content` field contains the actual content of the file as a string.

#### `metadata`

The `metadata` field contains a dictionary with the following schema:

| Key                    | Description                                       | Data Type | Example       |
|------------------------|---------------------------------------------------|-----------|---------------|
| `size`                 | The size of the file in bytes                     | Integer   | 12345         |
| `path`                 | The path of the file in the repository            | String    | "src/main.py" |
| `num_lines`            | The number of lines in the file                   | Integer   | 678           |
| `file_extension`       | The file extension                                | String    | ".py"         |
| `programming_language` | The detected programming language of the file     | String    | "Python"      |
| `is_likely_test_file`  | Flag indicating if the file is likely a test file | Boolean   | false         |

### Configuration

(`vdk config-help` is useful command to browse all config options of your installation of vdk)

| Name    | Description                              | (example)  Value               |
|---------|------------------------------------------|--------------------------------|
| git_url | URL of the Git repository to be cloned.	 | "https://github.com/user/repo" |


### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-control-api-auth 0.1.1431637373
Versatile Data Kit plugin library provides support for authentication.

# vdk-control-api-auth

<a href="https://pypistats.org/packages/vdk-control-api-auth" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-control-api-auth.svg" alt="monthly download count for vdk-control-api-auth"></a>

vdk-control-api-auth is a plugin library that implements authentication
utilities used by vdk-control-cli and other plugins.

## Usage

This is a library plugin, not a runnable plugin, and it is intended to be
used as a dependency for other plugins, which need to authenticate users.

To use the library within a plugin or another Versatile Data Kit component,
just import the `Authentication` class, and create an instance of it. The
different authentication flows require different parameters to be specified.

Once everything is done, in order to authenticate, call `.authenticate()` on
the `Authentication` instance.

Example Usage:
```python
from vdk.plugin.control_api_auth.authentication import Authentication

auth = Authentication(
    token="<oauth-api-token>",
    authorization_url="https://some-authorization-endpoint",
    auth_type="api-token",
)

auth.authenticate()   # authenticate

auth.read_access_token()   # fetch the cached access token
```

## Build and testing

In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it
vdk-duckdb 0.2.1431637373
DuckDB Plugin for VDK.

# duckdb

<a href="https://pypistats.org/packages/vdk-duckdb" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-duckdb.svg" alt="monthly download count for vdk-duckdb"></a>

DuckDB plugin for the Versatile Data Kit (VDK), which enables users to connect to and interact with DuckDB databases.
The purpose is to simplify data extraction, transformation, and loading tasks when working with DuckDB as a data source or destination

## Usage

```
pip install vdk-duckdb
```

### Configuration

Run `vdk config-help` to browse all available configuration options for your VDK installation.


### Example

#### Query Execution

You specify you want to use duckdb in the job config file
config.ini
```
[vdk]
db_default_type = duckdb
```

Then you can use it
```
    def run(job_input: IJobInput):
        job_input.execute_query("select 'Hi Duck!'")
```

#### Ingestion

This plugin allows users to ingest data to a DuckDB database, which can be preferable to inserting data manually as it automatically handles serializing, packaging and sending of the data asynchronously with configurable batching and throughput.
To do so, you must set the expected variables to connect to Greenplum, plus the following environment variable:

export VDK_INGEST_METHOD_DEFAULT=DUCKDB

Then, from inside the run function in a Python step, you can use the `send_object_for_ingestion` or `send_tabular_data_for_ingestion` methods to ingest your data.

### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-audit 0.1.1431637373
Versatile Data Kit SDK Audit plugin restricts forbidden operations.
Requires Python: >=3.8

## Versatile Data Kit Audit Plugin

<a href="https://pypistats.org/packages/vdk-audit" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-audit.svg" alt="monthly download count for vdk-audit"></a>

Visibility into the actions provides opportunities for test frameworks, logging
frameworks, and security tools to monitor and optionally limit actions taken by the
runtime.
This plugin provides an ability to audit and potentially limit user operations.
These operations are typically deep within the Python runtime or standard library, such
as dynamic code compilation, module imports or OS command invocations. In order to have a
better understanding of what exactly the job does, we will log all job operations.

### Usage

To use the plugin, just install it using

```bash
pip install vdk-audit
```
setup-trufflehog-gitleak.2 0.1
A package to install and use security tools like TruffleHog and Gitleaks.
Requires Python: >=3.6

# Setup Security Tools

This package installs OWASP ZAP, Gitleaks, and TruffleHog on a Windows machine using Chocolatey and pip.

## Setup a Package:

Install a package:
```bash
pip install security-package-tools
```

Install setuptools and wheel if you haven't already:

```bash
pip install setuptools wheel
```

## Usage

To use this package, install it locally and run the command:

```bash
setup-security-tools
```
setup-trufflehog-gitleak.1 0.6
A package to install and use security tools like TruffleHog and Gitleaks.
Requires Python: >=3.6

# Setup Security Tools

This package installs OWASP ZAP, Gitleaks, and TruffleHog on a Windows machine using Chocolatey and pip.

## Setup a Package:

Install a package:
```bash
pip install security-package-tools
```

Install setuptools and wheel if you haven't already:

```bash
pip install setuptools wheel
```

## Usage

To use this package, install it locally and run the command:

```bash
setup-security-tools
```