rcd-test 1.0.0
Module to manage different database

# Database Manager

Database Manager is a Python package for manipulating data with different databases.

## Installation

Make sure database_manager folder and decorator_manager exists under helpers. 

Set up .env as follow:
```python
#elasticsearch
ELASTICSEARCH_SCHEME=https
ELASTICSEARCH_PORT=9243
ELASTICSEARCH_HOST=x
ELASTICSEARCH_USER=x
ELASTICSEARCH_PASSWORD=x

#redshift
REDSHIFT_HOST=x
REDSHIFT_USER=x
REDSHIFT_PASSWORD=x
REDSHIFT_DB=x

#S3
S3_BUCKET_DEV=oip-openinnovationprogram-dev
S3_BUCKET_RAW=oip-openinnovationprogram-raw
S3_BUCKET_DATAMART=oip-openinnovationprogram-datamart
AWS_ACCESS_KEY_ID=x
AWS_SECRET_ACCESS_KEY=x
AWS_DEFAULT_REGION=eu-west-3
```

## Usage
### Elasticsearch
* index_json_bulk_parallel
```python
import os
import pandas as pd
from helpers import database_manager
# bulk indexing data to elasticsearch
database_manager.index_json_bulk_parallel(index_name="my_index_name", method="json", json_path=os.environ.get("DRUG_JSON_PATH"), keyword="") # with json file
database_manager.index_json_bulk_parallel(index_name="my_index_name", method="dataframe", pd_dataframe=pd.DataFrame()) # with pandas dataframe, pass your dataframe by pd_dataframe
```
* index_json_bulk
```python
import os
from helpers import database_manager
database_manager.index_json_bulk(json_path=os.environ.get("DRUG_JSON_PATH"), index_name="my_index_name", keyword="") # bulk indexing data to elasticsearch
```

* index_json
```python
import os
from helpers import database_manager
database_manager.index_json(json_path=os.environ.get("DRUG_JSON_PATH"), index_name="my_index_name", keyword="") # indexing data to elasticsearch
```

### Redshift
* send_to_redshift
```python
import os
import pandas as pd
from helpers import database_manager
database_manager.send_to_redshift(database="my_database", schema="test_schema", table="my_table_name", df=pd.DataFrame()) # fast way to send data to redshift&s3 without checking consistency
database_manager.send_to_redshift(database="my_database", schema="test_schema", table="my_table_name", df=pd.DataFrame(), check=True) # slow way to send data to redshift&s3 but will table structure
```
* read_from_redshift
```python
import os
from helpers import database_manager
database_manager.read_from_redshift(database="my_database", method="auto", schema="my_schema", table="my_table_name") # reading data from redshift with auto mode
database_manager.read_from_redshift(database="my_database", method="sql", sql_query='SELECT * FROM reference.jp__gender_dictionary__mhlw') # reading data from redshift with sql query
```
### S3
* upload_raw_s3
```python
from helpers import database_manager
database_manager.upload_raw_s3(local_folder_path="my_folder", uuid="my_uuid")
```
* download_raw_s3
```python
from helpers import database_manager
database_manager.download_raw_s3(local_folder_path="my_folder", uuid="my_uuid")
```
## Roadmap
* Redshift: Find package for calculating significant digits of precision.
* Redshift: Dynamically adding or deleting columns.
* Redshift: Transfer sqlalchemy core to sqlalchemy orm.


## Feedback
Any questions or suggestions?

Please contact package maintainer **yu.levern@realconsultingdata.com**