semvec/pypi-packages huggingface dataset:
dpk-proglang-select-transform-ray 0.2.0
Programming Language Selection Ray Transform
License: Apache-2.0
Requires Python: >=3.10

# Programming Language Select 

Please see the set of
[transform project conventions](../../../README.md)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This project enables the [python malware transform](../python) to be run in a Ray runtime.
Please see the [python project](../python) for details on the transform implementation and use.

## Configuration and Command Line Options

Transform configuration options are the same as the base python transform.

## Running

### Launched Command Line Options
In addition to those available to the transform as defined in [here](../python/README.md),
the set of
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.


### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/proglang_select_transform_ray.py using command line args
* `run-local-ray-sample` - runs src/proglang_select_local_ray.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-tokenization-transform-ray 0.2.0
Tokenization Transform for Ray
License: Apache-2.0
Requires Python: >=3.10

# Tokenization Transform for Ray
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This project wraps the [tokenization transform](../python) with a Ray runtime.

## Configuration and command line Options

Noop configuration and command line options are the same as for the base python transform. 

## Running

### Launched Command Line Options 
In addition to those available to the transform as defined in [here](../python/README.md),
the set of 
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/tokenization_transform_ray.py using command line args
* `run-local-sample` - runs src/tokenization_local_ray.py
* `run-s3-sample` - runs src/filter_s3_ray.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-noop-transform-spark 0.2.0
NOOP Spark Transform
License: Apache-2.0
Requires Python: >=3.10

# NOOP Spark Transform 
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This project wraps the [noop transform](../python) with a Ray runtime.

## Configuration and command line Options

Noop configuration and command line options are the same as for the base python transform.

## Running

### Launched Command Line Options
In addition to those available to the transform as defined in [here](../python/README.md),
the set of
[spark launcher](../../../../data-processing-lib/doc/spark-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/noop_transform.py using command line args
* `run-local-sample` - runs src/noop_local_spark.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
```shell
make run-cli-sample
...
```
Then
```shell
ls output
```
To see results of the transform.
dpk-noop-transform-ray 0.2.0
NOOP Ray Transform
License: Apache-2.0
Requires Python: >=3.10

# NOOP Ray Transform 
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This project wraps the [noop transform](../python) with a Ray runtime.

## Configuration and command line Options

Noop configuration and command line options are the same as for the base python transform. 

## Running

### Launched Command Line Options 
In addition to those available to the transform as defined in [here](../python/README.md),
the set of 
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/noop_transform.py using command line args
* `run-local-sample` - runs src/noop_local_ray.py
* `run-s3-sample` - runs src/noop_s3_ray.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-noop-transform-python 0.2.0
NOOP Python Transform
License: Apache-2.0
Requires Python: >=3.10

# NOOP Transform 
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This transform serves as a template for transform writers as it does
not perform any transformations on the input (i.e., a no-operation transform).
As such, it simply copies the input parquet files to the output directory.
It shows the basics of creating a simple 1:1 table transform.
It also implements a single configuration value to show how configuration
of the transform is implemented.

## Configuration and command line Options

The set of dictionary keys holding [NOOPTransform](src/noop_transform.py) 
configuration for values are as follows:

* _noop_sleep_sec_ - specifies the number of seconds to sleep during the call the 
the `transform()` method of `NOOPTransformation`.  This may be useful for
simulating real transform timings and as a way to limit I/O on an S3 endpoint..
* _noop_pwd_ - specifies a dummy password not included in metadata. Provided
as an example of metadata that we want to not include in logging.

## Running

### Launched Command Line Options 
The following command line arguments are available in addition to 
the options provided by 
the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md).
```
  --noop_sleep_sec NOOP_SLEEP_SEC
                        Sleep actor for a number of seconds while processing the data frame, before writing the file to COS
  --noop_pwd NOOP_PWD   A dummy password which should be filtered out of the metadata
```
These correspond to the configuration keys described above.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/noop_transform.py using command line args
* `run-local-sample` - runs src/noop_local.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-filter-transform-ray 0.2.0
Filter Transform for Ray
License: Apache-2.0
Requires Python: >=3.10

# Fltering Transform for Ray

Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This project wraps the [filter transform](../python) with a Ray runtime.

## Configuration and command line Options

Noop configuration and command line options are the same as for the base python transform. 

## Running

### Launched Command Line Options 
In addition to those available to the transform as defined in [here](../python/README.md),
the set of 
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/filter_transform.py using command line args
* `run-local-sample` - runs src/filter_local_ray.py
* `run-s3-sample` - runs src/filter_s3_ray.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-malware-transform-ray 0.2.0
Malware Ray Transform
License: Apache-2.0
Requires Python: >=3.10

# Malware Transform 
Please see the set of
[transform project conventions](../../../README.md#Transform-Project-Conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This project enables the [python malware transform](../python) to be run in a Ray runtime.
Please see the [python project](../python) for details on the transform implementation and use.

## Configuration and Command Line Options

Transform configuration options are the same as the base python transform.

## Running

### Launched Command Line Options 
In addition to those available to the transform as defined in [here](../python/README.md),
the set of
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/malware_transform.py using command line args
* `run-local-sample` - runs src/malware_local.py
* `run-local-ray-sample` - runs src/malware_local_ray.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-code-quality-transform-ray 0.2.0
Code Quality Ray Transform
License: Apache-2.0
Requires Python: >=3.10

# Code Quality 

Please see the set of
[transform project conventions](../../../README.md)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This project enables the [python malware transform](../python) to be run in a Ray runtime.
Please see the [python project](../python) for details on the transform implementation and use.

## Configuration and Command Line Options

Transform configuration options are the same as the base python transform.

## Running

### Launched Command Line Options
In addition to those available to the transform as defined in [here](../python/README.md),
the set of
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-ray-sample` - runs src/code_quality_transform.py using command line args
* `run-local-ray-sample` - runs src/code_quality_local_ray.py
* `run-s3-ray-sample` - runs src/code_quality_s3_ray.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.
dpk-code2parquet-transform-ray 0.2.0
code2parquet Ray Transform
License: Apache-2.0
Requires Python: >=3.10

# NOOP Ray Transform 
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary 
This project wraps the [code2parquet transform](../python) with a Ray runtime.

## Configuration and command line Options

code2parquet transform configuration and command line options are the same as for the base python transform. 

## Running

### Launched Command Line Options 
In addition to those available to the transform as defined in [here](../python/README.md),
the set of 
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/code2parquet_transform.py using command line args
* `run-local-sample` - runs src/code2parquet_local_ray.py
* `run-s3-sample` - runs src/code2parquet_s3_ray.py
    * Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
     and [here](https://min.io/docs/minio/linux/index.html) 
     and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.