near-lake-framework 0.1.3
Python Library to connect to the NEAR Lake S3 and stream the data
Requires Python: >=3.9

# NEAR Lake Framework for Python

Available in programming languages: [Rust](https://github.com/near/near-lake-framework-rs) | [Javascript](https://github.com/near/near-lake-framework-js) | **Python3**

NEAR Lake Framework is a small library companion to [NEAR Lake](https://github.com/near/near-lake). It allows you to build
your own indexer that subscribes to the stream of blocks from the NEAR Lake data source and create your own logic to process
the NEAR Protocol data.

[![PyPI version](https://badge.fury.io/py/near-lake-framework.svg)](https://badge.fury.io/py/near-lake-framework)
![MIT or Apache 2.0 licensed](https://img.shields.io/crates/l/near-lake-framework.svg)

---

[Official NEAR Lake Framework launch announcement](https://gov.near.org/t/announcement-near-lake-framework-brand-new-word-in-indexer-building-approach/17668) has been published on the NEAR Gov Forum
Greetings from the Data Platform Team! We are happy and proud to announce an MVP release of a brand new word in indexer building approach - NEAR Lake Framework.

---

## Example

```python3
import asyncio
import logging
import os

from near_lake_framework import LakeConfig, streamer, Network
from near_lake_framework.utils import fetch_latest_block

# Suppress warning logs from specific dependencies
logging.getLogger("near_lake_framework").setLevel(logging.INFO)


async def main():
    network = Network.TESTNET
    latest_final_block = fetch_latest_block(network=network)
    config = LakeConfig(
        network,
        start_block_height=latest_final_block,
        # These fields must be set!
        aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
        aws_secret_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    )

    stream_handle, streamer_messages_queue = streamer(config)
    while True:
        streamer_message = await streamer_messages_queue.get()
        print(
            f"Received Block #{streamer_message.block.header.height} from Lake Framework"
        )


if __name__ == "__main__":
    loop = asyncio.new_event_loop()
    loop.run_until_complete(main())
```

Try it yourself as follows
```shell
pip3 install -r requirements.txt
python3 example/run.py
```

## How to use

### Dependencies

Install `near-lake-framework`

```bash
$ pip3 install near-lake-framework
```

### Credentials

To be able to access the data from NEAR Lake you need to provide credentials. Please, see the [Credentials article](https://near-indexers.io/tutorials/lake/credentials)

## Configuration

Everything should be configured before the start of your indexer application via `LakeConfig` struct.

Available parameters:

- `network: Network` - provide network
- `s3_bucket_name: str` - provide the AWS S3 bucket name (`near-lake-testnet`, `near-lake-mainnet` or yours if you run your own NEAR Lake)
- `s3_region_name: str` - provide the region for AWS S3 bucket
- `start_block_height: BlockHeight` - block height to start the stream from
- `blocks_preload_pool_size: int` - provide the number of blocks to preload (default: 200)

## Cost estimates

**TL;DR** approximately $18.15 per month (for AWS S3 access, paid directly to AWS) for the reading of fresh blocks

Explanation:

Assuming NEAR Protocol produces accurately 1 block per second (which is really not, the average block production time is 1.3s). A full day consists of 86400 seconds, that's the max number of blocks that can be produced.

According the [Amazon S3 prices](https://aws.amazon.com/s3/pricing/?nc1=h_ls) `list` requests are charged for $0.005 per 1000 requests and `get` is charged for $0.0004 per 1000 requests.

Calculations (assuming we are following the tip of the network all the time):

```
86400 blocks per day * 5 requests for each block / 1000 requests * $0.0004 per 1k requests = $0.173 * 30 days = $5.19
```
**Note:** 5 requests for each block means we have 4 shards (1 file for common block data and 4 separate files for each shard)

And a number of `list` requests we need to perform for 30 days:

```
86400 blocks per day / 1000 requests * $0.005 per 1k list requests = $0.432 * 30 days = $12.96

$5.19 + $12.96 = $18.15
```

The price depends on the number of shards

## Real-life Example

Pagoda, a leading blockchain infrastructure company, utilizes the NEAR Lake Framework for Python in their [Gas Station Event Indexer](https://github.com/near/gas-station-event-indexer). This real-world implementation showcases the framework's capabilities in efficiently processing and indexing blockchain data.

## Publishing to PyPi

[Follow this guide to safely publish PyPi package](https://widdowquinn.github.io/coding/update-pypi-package/)