semvec/pypi-packages huggingface dataset:
data-minimization-tools 0.1.3
Pyhton library for data minimization tools.
License: MIT

# data minimization tools ![Python package](https://github.com/peng-data-minimization/minimizer/workflows/Python%20package/badge.svg)
Python library for data minimization tools.


## Installation

`pip install data-minimization-tools`

See on [PyPi](https://pypi.org/project/data-minimization-tools/).


## Documentation
[Read the Docs](https://data-minimization-tools.readthedocs.io/en/latest/)


## Development
Feel free to contribute. To install from source run `pip install .`.
zhongkui-core 0.1.0
zhongkui core utils package
License: MIT
Requires Python: >=3.7

# zhongkui-core

zhongkui core analysis package


## Installation

1. install exiftool

```
#!/bin/sh

# only test on ubuntu18.04

export EXIFTOOL=12.01
echo "Installing exiftool..." 
curl -Ls https://exiftool.org/Image-ExifTool-$EXIFTOOL.tar.gz > /tmp/exiftool.tar.gz 
cd /tmp 
tar xzf exiftool.tar.gz 
cd Image-ExifTool-$EXIFTOOL 
perl Makefile.PL 
make test 
make install 
```

2. install zhongkui-core

```
$ git clone git@git.kongkongss.com:zhongkui/zhongkui-core.git
$ cd zhongkui-core
$ pip install -e .
```

## Getting Started

```shell
>>> from zhongkui.core import File
>>> sample = File("tests/sample/pe_upx")
>>> print(sample.basicInfo())
>>>
{
    "name": "pe",
    "md5": "ff2a00e3d07afcf32a7459040bc9cc41",
    "sha1": "61e810f352866cc5f3184489c796fba4c21b99ec",
    "sha256": "fb12aec2553bd2567a82f18ca2e0710e8d72b22b1d2bdcf3a296e987ad3c398a",
    "fileType": "win32exe",
    "fileSize": 3901952,
    "isProbablyPacked": true
}
```


## Running the tests

```shell
$ cd zhongkui-core
$ pytest -s
```

## Changelog
[release Changelog](./CHANGELOG.md)


## Authors

* **kongkong Jiang** - *Initial work* - [jyker](https://git.kongkongss.com/jyker)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details
jyk-time 0.1.0
jyk time utils
License: MIT
Requires Python: >=3.7

# jyk-time

jyk time utils


## Installation
```
$ pip install jyk-time
```

## Running the tests

```shell
$ cd jyk-time
$ pytest -s
```

## Changelog

## 0.1.0
> 2020-07-15 release


## Authors

* **kongkong Jiang** - *Initial work* - [jyker](https://git.kongkongss.com/jyker)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details
kaggle-runner 0.2.1
Run kaggle kernels, for fast model prototyping.
License: MIT
Requires Python: >=3.6

[![Build Status](https://travis-ci.org/pennz/kaggle_runner.svg?branch=master)](https://travis-ci.org/pennz/kaggle_runner)
[![PyPI version](https://badge.fury.io/py/kaggle-runner.svg)](https://badge.fury.io/py/kaggle-runner)
[![Maintainability](https://api.codeclimate.com/v1/badges/979bc98e4acb59a5e1aa/maintainability)](https://codeclimate.com/github/pennz/kaggle_runner/maintainability)
[![pipeline status](https://gitlab.com/MrCue/kaggle_runner/badges/master/pipeline.svg)](https://gitlab.com/MrCue/kaggle_runner/-/commits/master)
[![coverage report](https://gitlab.com/MrCue/kaggle_runner/badges/master/coverage.svg)](https://gitlab.com/MrCue/kaggle_runner/-/commits/master)

[Generated doc website](https://pennz.github.io/kaggle_runner)

# kaggle_runner

Check main.py or test/test_coord.py for usage. It uses kaggle API to upload your script/notebook to kaggle servers and let the kernel run. And you will get running logs through message queue.

## AMQP
AMQP is used for logging. Its license needs mention.

## Example
Use this [kaggle competition](https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation) about Pneumothorax Segmentation as an example. To run the example, you will need a kaggle account and set the kaggle command line tool up. And you issues this command to let it run:

```sh
# install kaggle_runner, which will pull kaggle command line tool as the dependency
pip install kaggle_runner

# put your kaggle API token to the right place
cat > ~/.kaggle/kaggle.json <<EOF
{
  "username": "YOUR_KAGGLE_USER_NAME",
  "key": "YOUR_KAGGLE_API_ACCESS_TOKEN",
}
EOF

# kaggle_runner will use kaggle API to push the template kernel codes to kaggle server and wait message back
python -m kaggle_runner
```

A demo:

1. \#0 Left panel: tcpserver listen for reverse shells
1. \#1 Upper panel: Logs from interactive session to our tcpserver which receive logs
1. \#2 Second upper panel: AMQP logs received
1. \#3 Main panel: vim window
1. \#4 Right bottom panel: logged in reverse shell for commit session

[![asciicast](https://asciinema.org/a/vcLKH8MEkxv4WYEb9xxK8xBnU.svg)](https://asciinema.org/a/vcLKH8MEkxv4WYEb9xxK8xBnU)
icgen 0.3.0
 Image Classification Dataset Generator
License: MIT
Requires Python: >=3.6.1,<4.0.0

# ICGen

## Installation

```
pip install icgen
```

for a development installation see [CONTRIBUTING.md](CONTRIBUTING.md)


## Usage

### Sampling Datasets

```python
import icgen
dataset_generator = icgen.ICDatasetGenerator(
  data_path="datasets",
  min_resolution=16,
  max_resolution=512,
  max_log_res_deviation=1,  # Sample only 1 log resolution from the native one
  min_classes=2,
  max_classes=100,
  min_examples_per_class=20,
  max_examples_per_class=100_000,
)
dev_data, test_data, dataset_info = dataset_generator.get_dataset(
    dataset="cifar10", augment=True, download=True
)
```

The `augment` parameter controls whether the original dataset is modified.

Options only affect sampling with `augment=True` and the min max ranges do not filter datasets.

The data is left at the original resolution, so it can be resized under user control.
This is necessary to for example avoid resizing twice which can hurt performance.

You can also sample from a list of datasets
```python
dataset_generator.get_dataset(datasets=["cifar100", "emnist/balanced"], download=True)
```

We provide some lists of available datasets
```python
import icgen
icgen.DATASETS_TRAIN
icgen.DATASETS_VAL
icgen.DATASETS_TEST
icgen.DATASETS
```
or on the commandline you can get the names with
```
python -m icgen.dataset_names
```


### Downloading Datasets Before Execution

To download datasets ahead of time you can run

```
python -m icgen.download --data_path DATA_PATH --datasets D1 D2 D3
```

or directly download a complete group

```
python -m icgen.download --data_path DATA_PATH --dataset_group GROUP  # all, train, dev, test
```

Alternatively, you can also use the `download=True` flag of the `dataset_generator.get_dataset` function.


### Reconstructing and Distributing Tasks

In distributed applications it may be necessary to sample datasets on one machine and then use them on another one.
Conversely, for reproducibility it may be necessary to store the exact dataset which was used.
For these cases icgen uses a dataset identifier which uniquely identifies datasets.


## License

[MIT](LICENSE)