semvec/pypi-packages huggingface dataset:
cspframework 0.1.0
A CSP framework on Maya platform for algorithm deployment and inference.
Requires Python: >=3.6

## README的意义

README 文件通常是项目的第一个入口点。你应该通过 README 明确地告诉大家，为什么他们应该使用你的项目，以及安装和使用的方法。

如果在仅仅看文档而不看代码的情况下就可以使用你的项目，该文档就完成了。 这个非常重要，因为这将使项目的文档接口与其内部实现分开，只要接口保持不变，就可以自由更改项目的内部结构。 

**文档，而不是代码定义了项目的使用方式。**

一个规范的README文档能减少用户检索信息的时间。

## 标准 README

一个标准的README文件应当至少包含以下的内容：

- 项目背景：说明创建本项目的背景与动机，创建本项目试图解决的问题 
- 安装方法：说明如何快速上手使用该项目
- 使用方法：列出本项目能够提供的功能以及使用这些功能的方法
- 文档：现阶段antcode鼓励用户使用语雀组织项目文档，在README上应当放入项目的语雀文档链接

## 附加内容

视项目的实际情况，同样也应该包含以下内容：

- 项目特性：说明本项目相较于其他同类项目所具有的特性
- 兼容环境：说明本项目能够在什么平台上运行
- 使用示例：展示一些使用本项目的小demo
- 主要项目负责人：使用“@”标注出本项目的主要负责人，方便项目的用户沟通
- 参与贡献的方式：规定好其他用户参与本项目并贡献代码的方式
- 项目的参与者：列出项目主要的参与人
- 已知用户：列出已经在生产环境中使用了本项目的全部或部分组件的公司或组织
- 赞助者：列出为本项目提供赞助的用户
mbench 0.0.11
Simple Benchmarking Library.
Requires Python: >=3.10

# mbench

[![PyPI - Version](https://img.shields.io/pypi/v/mbench.svg)](https://pypi.org/project/mbench)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mbench.svg)](https://pypi.org/project/mbench)

-----

Simple benchmarking tool for a module, function, or block of code.


## Installation

```console
pip install mbench
```

## Usage

```python
from mbench import profileme
profileme()

def some_function():
    print("Hello")

some_function()
```
```console
        Profile Information for         
     __main__.test_get_object_speed     
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃           Metric ┃ Value             ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│         Duration │ 1.461318 seconds  │
│         CPU time │ 24.613764 seconds │
│     Memory usage │ 250.56 MB         │
│        GPU usage │ 0.00 B            │
│       GPU usages │                   │
│        I/O usage │ 0.00 B            │
│     Avg Duration │ 0.292264 seconds  │
│     Avg CPU time │ 4.922753 seconds  │
│ Avg Memory usage │ 50.11 MB          │
│    Avg GPU usage │ 0.00 B            │
│   Avg GPU usages │                   │
│    Avg I/O usage │ 0.00 B            │
│      Total calls │ 5                 │
└──────────────────┴───────────────────┘
```
### As a Decorator

```python
from mbench import profile
@profile
def some_function():
    print("Hello")
```

### As a Context Manager
```python

from mbench import profiling
with profiling:
  run_anything()
```

## _when_ calling

Functions you want to profile must

1. Be _defined_ in the same module that the `profileme` function is being called.
2. Be called after `profileme(when="calling")` is called.

## _when_ called

Functions you want to profile must

1. Be _called_ in the same module that the `profileme` function is being called.
2. Be called after `profileme(when="called")` is called.

## Docs
```python
profileme(when: Literal['called', 'calling'] = 'called')
    Profile all functions in a module. Set `when` to 'calling' to profile only the functions called by the target module.
```
## License

`mbench` is distributed under the terms of the [MIT License](LICENSE).
django-shell-extended 1.0.0
An extended Django shell
Requires Python: >=3.6

# django-shell-extended

Provides a `sh` management-command that can be customized via django settings.

## Quick start
1. Install with `pip install django-shell-extended`
2. Add `"django_shell_extended"` to your `INSTALLED_APPS` setting like this:
    ```python
    INSTALLED_APPS = [
        # ...
        "django_shell_extended",
    ]
    ```
3. (Optional) Change the settings described in the next section in your `settings.py`
4. Start the shell via `python manage.py sh`

## Available settings
```python
# The Path to a custom ipython directory.
# default: None
DJANGO_SH_IPYTHON_DIR = "/some/path/.ipython/"

# Tries to autodetect django apps with models and imports them as `from my_app.models import *`
# default: True
DJANGO_SH_AUTO_IMPORT_MODELS = True

# Defines additional modules that should be imported.
# Should be a tuple or list containing entries like `(module, imports)`.
# default: ()
DJANGO_SH_IMPORT_MODULES = (
    ('datetime', ('datetime', 'timedelta')),  # -> from datetime import datetime, timedelta
    ('re', None),                             # -> import re
    ('decimal', 'Decimal'),                   # -> from decimal import Decimal
    ('foo.bar', '*')                          # -> from foo.bar import *
)

# A function that adds command line arguments to the `sh` command.
# default: None
def DJANGO_SH_HOOK_ADD_ARGUMENTS(parser: CommandParser) -> None:
    parser.add_argument(...)


# A function that is called after importing models and modules.
# Can be used to further extend the `imported_objects` name space or to check command arguments. Example::
# default: None
def DJANGO_SH_HOOK_GET_IMPORTED_OBJECTS(imported_objects: dict, options: dict) -> None:
    multiplier = int(options.get('multiplier', 0))
    imported_objects['multiply'] = lambda x: x * multiplier


# Can be used to alter the command provided to `python manage.py sh -c '<command>'`
# default: None
def DJANGO_SH_HOOK_MODIFY_COMMAND(command: str) -> str:
    return f'start=time.time()\\n{command}\\nprint("Command took", time.time() - start, "seconds")'
```
fy-polyfit 1.2.2
polyfit data use linear and Quadratic
Requires Python: >=3.8


# polyfit 一次/二次拟合工具


## 说明

本程序主要功能为读取待拟合的数据文件，文件格式可以为.txt 或者.csv格式，以空格分隔，无头部。
通过命令行 `polyfit` [filename] 执行,如果未指定文件名，则默认寻找当前目录下的 `data.txt`

数据样本为N行两列的数据，第一列为x, 第二列为Y,
实例数据如下：

```text

   0.134	2.79
   0.160	3.39
   0.195	4.09
   0.244	5.19
   0.313	6.59
```

## 命令行执行
调用 `polyfit [data.txt]`

执行结果如下：

![拟合图形](https://github.com/archeno/polyfit/blob/main/images/fitting.png?raw=true)
vdk-test-utils 0.2.1431637373
Provides utilities for testing Versatile Data Kit SDK plugins.

<a href="https://pypistats.org/packages/vdk-test-utils" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-test-utils.svg" alt="monthly download count for vdk-test-utils"></a>

This plugin provides utility tools used for the testing of vdk-core and vdk-core plugins.

# Usage

### CliEntryBasedTestRunner

CliEntryBasedTestRunner provides an easy way to create unit or functional tests for your plugins in isolated environment.
Similarly this can be used to unit tests data jobs.

For example , let's say we want to test newly developed AtlasLineagePlugin

```python
db_plugin = AtlasLineagePlugin()
# we pass the plugin to the runner that will make sure to register it
runner = CliEntryBasedTestRunner(db_plugin)

# and then invoke the command we want to test -
# in this case data job run
result: Result = runner.invoke(
    [ "run", util_funcs.jobs_path_from_caller_directory("simple-create-insert") ]
)

cli_assert_equal(0, result)
assert 'expected_schema' in result.output
```

### Testing plugins

Some useful plugins that can be used during testing can be seen in util_plugins.
vdk-snowflake 0.2.1431637373
Versatile Data Kit SDK plugin provides support for snowflake databases.

# Versatile Data Kit Plugin for Snowflake Support

<a href="https://pypistats.org/packages/vdk-snowflake" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-snowflake.svg" alt="monthly download count for vdk-snowflake"></a>

This plugin provides functionality, used by the Versatile Data Kit
to interact with a Snowflake instance. Users of the plugin can connect to
Snowflake, and execute queries against it.

### Instalation

To install the plugin, open a terminal and type
```bash
pip install --upgrade vdk-snowflake
```

### Usage

In order to use the plugin, once it has been installed, there are a few environment variables that need to be set.

The following environment variables are required:

```bash
\\ Set default database plugin to be used
VDK_DB_DEFAULT_TYPE=SNOWFLAKE

\\ Set the username that is to be used.
VDK_SNOWFLAKE_USER=<some_username>

\\ Set the password that is to be used.
VDK_SNOWFLAKE_PASSWORD=<user_password>

\\ Set the account name that is to be used.
\\ NOTE: Do NOT include the '.snoflakecomputing.com' part of the account name.
VDK_SNOWFLAKE_ACCOUNT=<account_name_provided_by_snowflake>
```

The following environment variables are optional, and can be overwritten from within the data job.

```bash
\\ The default warehouse to be used.
VDK_SNOWFLAKE_WAREHOUSE=<default_warehouse>

\\ The default database to be used.
VDK_SNOWFLAKE_DATABASE=<default_database>

\\ The default database schema to be used.
VDK_SNOWFLAKE_SCHEMA=<default_schema>
```

After configuring the environment variables, to use the plugin, create a data job, and add the queries you want to execute in `.sql` files. When the job executed, Versatile Data Kit will execute the queries in the order specified in the [User Guide](https://github.com/vmware/versatile-data-kit/wiki/User-Guide#data-job-steps).
vdk-storage 0.1.1431637373
Library for access to different managed storages

# storage

Library for access to different managed storages


## Usage

```
pip install vdk-storage
```

### Configuration

(`vdk config-help` is useful command to browse all config options of your installation of vdk)

| Name | Description | (example)  Value |
|---|---|---|
| dummy_config_key | Dummy configuration | "Dummy" |

### Example

TODO

### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-smarter 0.1.1431637373
Making VDK smarter by employing ML/AI.

# VDK Smarter

<a href="https://pypistats.org/packages/vdk-smarter" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-smarter.svg" alt="monthly download count for vdk-smarter"></a>

Making VDK smarter by employing ML/AI.



## Usage

```
pip install vdk-smarter
```

### Configuration

(`vdk config-help` is useful command to browse all config options of your installation of vdk)


### Example

TODO# VDK Smarter

Making VDK smarter by employing ML/AI.



## Usage

```
pip install vdk-smarter
```

### Configuration

(`vdk config-help` and search for configuration starting with "openai")

### Example

By default reviews are disabled since they are expensive.

To enable you need to set `openai_review_enabled` to `true` in the configuration and `openai_api_key`.
You can see `vdk config-help` (search for configuration with openai prefix) for more information.

Once enabled on vdk run each query statement executed will be also reviewed and scored.

```
vdk run example
```
```bash
Query:
CREATE TABLE IF NOT EXISTS super_collider.example_table(
   vc_id STRING,
   esx_id STRING,
   vm_count INT
) STORED AS PARQUET;


Review:
  {
    "score": 5,
    "review": "No further changes needed. The query is efficient, readable, and follows best practices.
               There are no potential errors, optimization, or security vulnerabilities."}

...
Query:
CREATE TABLE IF NOT EXISTS super_collider.example_fact_snapshot(
                  vc_id STRING,
                  esx_count BIGINT,
                  vm_count BIGINT
) STORED AS PARQUET;


Review:
{"score": 4, "review": "The query is well-structured and follows best practices.
                        However, there is an opportunity to improve its readability by
                        adding comments to explain the purpose of the query  and the meaning of the parameters.
                        Additionally, there is potential to optimize the query by providing more precise column types."}
```

At the end a report is generated `queries_reviews_report.md` with all the queries and their reviews.

### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-singer 0.1.1431637373
The plugin provides seamless configuration and execution of Singer Taps and Targets.

# singer

<a href="https://pypistats.org/packages/vdk-singer" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-singer.svg" alt="monthly download count for vdk-singer"></a>

The vdk-singer plugin provides an easy way to integrate Singer Taps as data sources within the Versatile Data Kit (VDK).
This allows you to pull data from various external systems that have Singer Taps available and use them seamlessly
within your VDK pipelines.




## Usage

```
pip install vdk-singer
```

### Configuration

(`vdk config-help` is useful command to browse all config options of your installation of vdk)


You can configure the Singer data source via the SingerDataSourceConfiguration class. The configuration options include:

* tap_name: The name of the Singer Tap you are using.
* tap_config: A dictionary containing configuration specific to the Singer Tap.
* tap_auto_discover_schema: A boolean to indicate whether to auto-discover the schema.

```python
config = SingerDataSourceConfiguration(
    tap_name="tap-gitlab",
    tap_config={
        "api_url": "https://gitlab.com/api/v4",
        "private_token": "your_token_here",
        # ... other tap specific configurations
    },
    tap_auto_discover_schema=True,

```

### Example

This will change as we will introduce more user frinedly way of defining sources but currently it looks like this:

```python
from vdk.api.job_input import IJobInput
from vdk.internal.builtin_plugins.ingestion.source.factory import SingletonDataSourceFactory
from vdk.plugin.singer.singer_data_source import SingerDataSourceConfiguration

def run(job_input: IJobInput):
    data_source = SingletonDataSourceFactory().create_data_source("singer-tap")
    config = SingerDataSourceConfiguration(
        tap_name="tap-gitlab",
        tap_config={
            "api_url": "https://gitlab.com/api/v4",
            "private_token": "your_token_here",
            # ... other configurations
        },
        tap_auto_discover_schema=True,
    )
    data_source.connect(config, None)
    # ... rest of the job logic

```

#### List all likely available taps

```shell
vdk singer --list-taps
```

### Build and testing

```
pip install -r requirements.txt
pip install -e .
pytest
```

In VDK repo [../build-plugin.sh](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/build-plugin.sh) script can be used also.


#### Note about the CICD:

.plugin-ci.yaml is needed only for plugins part of [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins).

The CI/CD is separated in two stages, a build stage and a release stage.
The build stage is made up of a few jobs, all which inherit from the same
job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
They run according to rules, which are ordered in a way such that changes to a
plugin's directory trigger the plugin CI, but changes to a different plugin does not.
vdk-properties-fs 0.0.1431637373
Versatile Data Kit SDK plugin provides support for Properties API client that uses local FS storage.

<a href="https://pypistats.org/packages/vdk-properties-fs" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-properties-fs.svg" alt="monthly download count for vdk-properties-fs"></a>

This plugin allows vdk-core to read/write properties on the local FS. Mainly for development purposes,
to simplify a use-case with local Properties API usage, that would otherwise require a Control Service instance prerequisite.
For example, to quickly demo a data job that uses a secret, so that secret is quickly stored on the presenter's local FS.

# Usage

Run
```bash
pip install vdk-properties-fs
```

After this, data jobs will be able to read and write properties stored on the local file system.

For example

```python
    def run(job_input: IJobInput):
        my_props = {"key": "value"}
        job_input.set_all_properties(my_props)
        assert job_input.get_all_properties() == my_props
        assert job_input.get_property("key") == "value"
```

# Configuration

To enable the plugin, set "PROPERTIES_DEFAULT_TYPE" to "fs-properties-client".
Run vdk config-help - search for those prefixed with "FS_PROPERTIES_" to see what configuration options are available.

# Testing

Testing this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-properties-fs/requirements.txt

Run
```bash
pip install -r requirements.txt
```
vdk-server 0.1.1431637373
Versatile Data Kit SDK plugin that facilitates the installation of a local Control Service.

## Versatile Data Kit Server Plugin

<a href="https://pypistats.org/packages/vdk-server" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-server.svg" alt="monthly download count for vdk-server"></a>

This plugin facilitates installation of the Control Service in a local [Kind](https://kind.sigs.k8s.io/) cluster.

### Usage

To use the plugin, just install it using:
```bash
pip install vdk-server
```

To install the Control Service:
```bash
vdk server --install
```

To remove revious installation:
```bash
vdk server --uninstall
```
vdk-plugin-control-cli 0.1.1431637373
Versatile Data Kit SDK plugin exposing CLI commands for managing the lifecycle of a Data Jobs.

<a href="https://pypistats.org/packages/vdk-plugin-control-cli" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-plugin-control-cli.svg" alt="monthly download count for vdk-plugin-control-cli"></a>

This plugin allows vdk-core to access vdk-control-cli functionality.

Upon installing it enhances vdk with CLI commands for data job lifecycle management available from vdk-control-cli package

It will also install properties plugin and switch Properties in vdk to use Control Service Properties API.
By default no Properties backend is used otherwise.

# Usage


```bash
pip install vdk-plugin-control-cli
```

### New CLI commands

Then on the vdk CLI you should be able to see the new commands. Run
```bash
vdk --help
```
and you will see extra commands being added:
```
Commands:
  ...
  login                  Authentication against the Control Service.
  logout                 Logout the user from the Data Jobs Service by...
  create                 Creates a new data job in cloud and locally.
  delete                 Deletes a data job from the cloud.
  deploy                 Deploys a data job.
  ...
```

And you can use them from the same CLI `vdk`

### New properties backend client registered

You can now access remote properties from Control Service Properties API
Store state, configuration or secrets there using CLI or Python JobInput API

For example, let's store some api-uri

```bash
vdk properties --set 'api-uri' 'http://cool.cool.api.com'
```

and then we can use it in our data job

```python
def run(job_input: IJobInput):
    uri = job_input.get_property('api-uri')
    print(requests.get(uri))
```

# Build

The easiest way is to use the ../build-plugin.sh helper script

```bash
../build-plugin.sh
```
vdk-notebook 0.1.1431637373
A VDK plugin for working with notebooks
Requires Python: ~=3.7

# vdk-notebook

<a href="https://pypistats.org/packages/vdk-notebook" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-notebook.svg" alt="monthly download count for vdk-notebook"></a>

A new VDK plugin which supports running data jobs which consists of .ipynb files.
You can see [VDK Jupyter Integration VEP](https://github.com/vmware/versatile-data-kit/blob/main/specs/vep-994-jupyter-notebook-integration/README.md) for more information.


## Usage
To install the plugin you need to run:
```
pip install vdk-notebook
```

### Configuration
No specific notebook configurations are needed currently.

### Example
Here is a directory structure which consist of one parent directory "parent-dir" and two subdirectories.
One of the subdirectories is a data job("example-job"). This is how the structure looks like:
```
parent-dir
└── example-job
    ├── notebook.ipynb
    ├── requirements.txt
    └── config.ini
└── some-other-dir
```
To run the data job from the parent-dir we need to use:

```
vdk run example-job
```

- Each cell which has tag "vdk" is executed
- There could be those types of VDK Cells:
  - VDK Python Cell. Those are normal Python cells tagged with "vdk"
  - VDK SQL Cell. Those are cells marked with `%%vdksql` magic and tagged with "vdk"
  - VDK Ingest Cell. Those are celles marked with `%%vdkingest` magic.

For more information and examples see [vdk-ipython](../vdk-ipython/README.md) documentation

### Testing
Testing this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-notebook/requirements.txt

Run
```
pip install -r requirements.txt
```

## Architecture
See the architecture of the plugin in the "Detailed design" section of [the VEP](https://github.com/vmware/versatile-data-kit/blob/main/specs/vep-994-jupyter-notebook-integration/README.md).
vdk-lineage-model 0.0.1431637373
VDK Lineage Model plugin defines common lineage model and classes used for managing lineageinformation in other VDK plugins.

# VDK Lineage Model

<a href="https://pypistats.org/packages/vdk-lineage-model" alt="Monthly Downloads">
        <img src="https://img.shields.io/pypi/dm/vdk-lineage-model.svg" alt="monthly download count for vdk-lineage-model"></a>

VDK Lineage Model plugin aims to abstract emitting lineage data from VDK data jobs, so that different lineage loggers
can be configured at run time in any plugin that supports emitting lineage data.
The plugin describes the lineage data model and an interface for loggers that are responsible to send the lineage
information to the desired system.

At POC level currently. Breaking changes should be expected.

TODOs:
 - Describe Non-SQL lineage (ingest, load data,etc)
 - Extend support for all query types