semvec/pypi-packages huggingface dataset:
cryogen 0.2.5
helps you preserve an ethereum dataset
License: MIT
Requires Python: >=3.10

# cryogen

helps you preserve the ethereum dataset fresh, fast and small

## install

```shell
pip install cryogen
```

## features

intelligently consolidates cryo-extracted datasets, cutting the number of chunks by 400-1000x.

offers fast in-place conversion that reduces disk footprint by 2x and increases query performance.

keeps the dataset fresh so you can always come back to up-to-date data.

## usage

### `cryogen collect <dataset>`

collect or update a cryo dataset.

cryogen uses 1000 block batches with zstd -3 compression. the gaps are filled automatically. the dataset can be up to 1000 blocks behind head beacuse the align option is used.

```shell
# you can also specify data_dir using CRYO_DATA_DIR env var
cryogen collect contracts --data-dir ~/cryo_data

# collect a block range, same format as cryo
cryogen collect traces --blocks 17000000:
```

### `cryogen consolidate <dataset>`

consolidate a dataset in-place.

this command will merge parquets into larger files covering 1e4, 1e5, 1e6 blocks. smaller files are not touched until a larger contiguous block can be formed. the worst case for this algorithm is 17 + 9 + 9 + 9 = 44 files at block 17,999,000.

starting from v0.2.0 cryogen also reduces the number of row groups. it creates a row group when an uncompressed size of a batch reaches 32 mb. this row group size provides optimal performance without imposing higher memory requirements that could lead to heavy swapping.

```shell
cryogen consolidate contracts

# test the feature without overwriting the dataset
cryogen consolidate contraces --no-inplace
```

note that after consolidating with cryogen, you should use cryogen to update the dataset. cryo won't recognize larger chunks and would attempt to collect the already merged and deleted smaller chunks, outputting duplicate data in the dataset.

### `cryogen watch <dataset>`

combines `collect` and `consolidate` command.

keep it running and it will update the dataset periodically.

```shell
# refresh every 4 hours
cryogen watch contracts --interval 14400
```

### `cryogen info <dataset>`

collects info about a parquet dataset.

```shell
cryogen info contracts
# {'num_rows': 62466632, 'files': 38, 'row_groups': 17984, 'total_compressed_size': 7850356027, 'total_uncompressed_size': 29236070746, 'elapsed': 0.747}
```
MongoDB-Connect 0.0.5
A python package for connecting with database.
License: MIT
Requires Python: >=3.7

# requirements_dev.txt we use for the testing
It makes it easier to install and manage dependencies for development and testing, separate from the dependencies required for production.

# difference between requirements_dev.txt and requirements.txt

requirements.txt is used to specify the dependencies required to run the production code of a Python project, while requirements_dev.txt is used to specify the dependencies required for development and testing purposes.

# tox.ini
We use if for the testing in the python package testing against different version of the python 

## how tox works tox enviornment creation
1. Install depedencies and packages 
2. Run commands
3. Its a combination of the (virtualenvwrapper and makefile)
4. It creates a .tox


# pyproject.toml
it is being used for configuration the python project it is a alternative of the setup.cfg file. its containts configuration related to the build system
such as the build tool used package name version author license and dependencies

# setup.cfg
In summary, setup.cfg is used by setuptools to configure the packaging and installation of a Python projec

# Testing python application
*types of testing*
1. Automated testing 
2. Manual testing

*Mode of testing*
1. Unit testing
2. Integration tests

*Testing frameworks*

1. pytest
2. unittest
3. robotframework
4. selenium
5. behave
6. doctest

# check with the code style formatting and syntax(coding standard)

1. pylint
2. flake8(it is best because it containt 3 library pylint pycodestyle mccabe)
3. pycodestyle