semvec/pypi-packages huggingface dataset:
pyhouse 0.0.13
Lighthouse for Python: a package facilitating the creation of data pipelines.
License: Apache-2.0
Keywords: data lake engineering pipeline catalog cloud

# pyhouse

This is a port of [Lighthouse](https://github.com/datamindedbe/lighthouse), 
a library written in Scala, that facilitates the creation of data pipelines that
are based on [Apache Spark](https://spark.apache.org/). It also comes with some
related convenience functions, like integrations to the AWS parameter store.

This port is targeted at Python and PySpark. It is not an exact port of the 
Scala code: we add what we need as we go along.

## Usage

One of this library’s main usages is to build a class-based data catalog, that 
supports chaining of sources. For example, if you had a dataset in a text file
that needed to be transformed (clean, derive some statistic, …) then you could
write this as such:

```python
from pyhouse.datalake.file_system_data_link import FileSystemDataLink

link = FileSystemDataLink(
    environment="dev",
    session = get_spark(),
    path = "s3://bucket-foo/file-bar.csv",
    format="csv",
    savemode="errorifexists",
    partitioned_by=("some-key", "another-key"),
    options={"header": True, "sep": "\t"}
)

link.read().groupBy("client").count().show()
```

The advantage of such data links becomes clear when there are multiple of them
that are combined in a module (the “catalog”): there would be one source of 
truth that many scripts can refer to. Hardcoded paths scattered across scripts 
would be a thing of the past.
pytorch-dp 0.1b1
Train PyTorch models with Differential Privacy
License: Apache-2.0
Requires Python: >=3.6

# pytorch-dp: Train PyTorch models with Differential Privacy

[![CircleCI](https://circleci.com/gh/facebookresearch/pytorch-dp.svg?style=svg&circle-token=247aedbb579492ef94652ea93bbee4e169457795)](https://circleci.com/gh/facebookresearch/pytorch-dp)

pytorch-dp is a library that enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance and allows the client to online track the privacy budget expended at any given moment.

*PyTorch Privacy is currently a preview beta and under active development!*

### Target audience
This code release is aimed at two target audiences:
1. ML practicioners will find this code a gentle introduction to training a model with differential privacy as it requires minimal code changes.
2. Differential Privacy scientists will find this code easy to experiment and tinker with, allowing them to focus on what matters.


## Installation
pip:
```bash
pip install pytorch-dp
```

From source:
```bash
git clone https://github.com/facebookresearch/pytorch-dp.git
cd pytorch-dp
pip install -e .
```

## Getting started
To train your model with differential privacy, all you need to do is to declare a PrivacyEngine and attach it to your optimizer before running, eg:

```python
model = Net()
optimizer = SGD(model.parameters(), lr=0.05)
privacy_engine = PrivacyEngine(
    model,
    batch_size,
    sample_size,
    alphas=[1, 10, 100],
    noise_multiplier=1.3,
    max_grad_norm=1.0,
)
privacy_engine.attach(optimizer)
# Now it's business as usual
```
The [MNIST example](examples/mnist.py) contains an end to end run.

## Contributing
See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out.

## References

* [Mironov, Ilya. "Rényi differential privacy." 2017 IEEE 30th Computer Security Foundations Symposium (CSF). IEEE, 2017.](https://arxiv.org/abs/1702.07476)
* [Abadi, Martin, et al. "Deep learning with differential privacy." Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2016.](https://arxiv.org/abs/1607.00133)
* [Mironov, Ilya, Kunal Talwar, and Li Zhang. "R\'enyi Differential Privacy of the Sampled Gaussian Mechanism." arXiv preprint arXiv:1908.10530 (2019).](https://arxiv.org/abs/1908.10530)
* [Goodfellow, Ian. "Efficient per-example gradient computations." arXiv preprint arXiv:1510.01799 (2015).](https://arxiv.org/abs/1510.01799)
* [McMahan, H. Brendan, and Galen Andrew. "A general approach to adding differential privacy to iterative training procedures." arXiv preprint arXiv:1812.06210 (2018).](https://arxiv.org/abs/1812.06210)

## License
This code is released under Apache 2.0, as found in the [LICENSE](LICENSE) file.