semvec/pypi-packages huggingface dataset:
datalake-utils 2.2.1
Manipulate data on Amazon S3 using Apache Hadoop filesystem compability
Requires Python: >=3.7

# Data Lake Utility
Package to manipulate data from/into Amazon S3 using partitions compatible with Apache Hadoop filesystem.
At this moment, this package was conceived to handle JSON and Parquet formats. That being said, it expects a Pandas DataFrame.

Data will be written into Amazon S3 as a multi-line JSON or Apache Parquet, compressed as GZIP.
## Features
Convert list of dictionaries...
```python
[
    {"brand": "Ford","model": "Mustang","year": 1965},
    {"brand": "Pontiac","model": "GTO","year": 1964},
    {"brand": "Lamborghini","model": "Miura","year": 1966}
]
```
...to multi-line JSON compressed as GZIP
```text
{"brand": "Ford","model": "Mustang","year": 1965}
{"brand": "Pontiac","model": "GTO","year": 1964}
{"brand": "Lamborghini","model": "Miura","year": 1966}
```

...or to Apache Parquet compressed as GZIP


## How to use
This package does not slice data into partitions at the moment. You must handle slicing of data to write into partitions desired.

Check this example below:
```python
from datalake_utils.utils import DataLake
import pandas

data = [
    {
        "brand": "Ford",
        "model": "Mustang",
        "year": 1965
    },
    {
        "brand": "Pontiac",
        "model": "GTO",
        "year": 1964
    },
    {
        "brand": "Lamborghini",
        "model": "Miura",
        "year": 1966
    }
]

datalake = DataLake(
    bucket_name="vehicles",
    schema="motor_vehicles",
    table="cars",
    partitions=[
        {
            "key": "decade",
            "value": "1960s"
        }
    ],
)

datalake.append_to_s3(data=pandas.DataFrame(data), file_format="json")
```

It will create an object into Amazon S3 with the following structure:
```
motor_vehicles/
`-- cars/
    `-- decade=1960s/
        `-- 2c0fea6c-444e-11ed-969f-acde48001122.json.gz
```

To read all files from partition, do:
```python
retrieved_data = datalake.read_from_s3(file_format="json")
```
`retrieved_data` is a Pandas DataFrame object. It is not possible to read a specific file using this function.


To convert a Pandas DataFrame into a list of tuples containing a single JSON:
```python
datalake.df_to_tuples(data=retrieved_data)
```


To delete all files from partition, do:
```python
datalake.delete_from_s3()
```
Ngoto 0.0.39
Ngoto module
Keywords: python,harryludemann,osint,pentester,ngoto


# Ngoto
[![](https://github.com/HarryLudemann/Ngoto/workflows/pytests/badge.svg)]()
[![Maintainability](https://api.codeclimate.com/v1/badges/08e4dc1f109aaa6c4f75/maintainability)](https://codeclimate.com/github/HarryLudemann/Ngoto/maintainability)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/f50d18ce111d4faf99ff411b5129e920)](https://www.codacy.com/gh/HarryLudemann/Ngoto/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=HarryLudemann/Ngoto&amp;utm_campaign=Badge_Grade)
![Supported Python versions](https://img.shields.io/badge/python-3.8+-blue.svg)
[![version-1.3](https://img.shields.io/badge/version-0.0.33-blue)](https://github.com/Datalux/Osintgram/releases/tag/1.3)
[![platforms](https://img.shields.io/badge/platform-windows%20%7C%20linux-blue)](https://github.com/loseys/Oblivion/)


# Warning :warning:

<p align="center"><b>This tool is solely for educational purposes. Developer will not be responsible for any misuse of the tool</b></p>    

# Setup:
## Using as Command line tool:
#### 1. Clone Repo:
```
git clone https://github.com/HarryLudemann/Ngoto
```

#### 2. Install Required Modules:
Move into downloaded Ngoto folder then run:
```
pip install -r requirements.txt
```
or
```
pip3 install -r requirements.txt
```

#### 4. Run
Run 'python/python3 main.py' script:
```
python main.py
```
or
```
python3 main.py
```
Which will bring you to the following:

![](.github/LaunchScreen.png)

# Example:

```python
from ngoto import plugin, command, task, Ngoto


class Basic():
    @plugin(name='Tester', desc='Tester Plugin', folder='Random')
    def tester(self, logger):
        logger.info(f'Plugin ran', program='Test')

    @command(name='test', aliases=['t'], desc='Tests command')
    def test(self, pos, options, logger):
        logger.info(f'Command ran', program='Test')

    @task(name='TaskTest', delay=30, id='test', os=['Windows'],
          desc="Tests task creation")
    def testing(self):
        return ['Task ran', 'Test']


ngoto = Ngoto()
ngoto.load_cogs([Basic()])
ngoto.main()
```
hyc-utils 0.5.33
Commonly used tools across my own personal projects

# hyc-utils
A personal python package containing commonly used utils within my projects

# Install
`pip install hyc-utils`

If you're developing the package, then clone/fork the package, and in the top level directory of the repository do

`pip install -r requirements.txt`

Since this package requires pytorch, if you're on a linux system and want a cpu-only version of pytorch, you can instead do

`pip install -r requirements_linux_cpu.txt`

Note: If you're on M1 Mac, there's a problem with pytables version 3.8.0 (as of writing) that you might encounter during install in which it complains it cannot find hdf5 install. If that is the case, then do

`export HDF5_DIR=/opt/homebrew/opt/hdf5`

you can put this line in your `~/.bashrc` file for convenience.