stockdataservice 0.1.1
A Python package for handling a stock data service of HTT
Requires Python: >=3.8,<4.0

# Stock Data Service v2

![Stock Data Service Logo](https://www.trend-impact-service.nl/static/StockdataInfra.png)

## Overview

Welcome to the Stock Data Service! This repository provides a comprehensive stock data service that allows users to access and analyze stock market data easily. 

## Features

- Updates Historical EOD price data on the US, EU and Asian session
- Executes large correlation researches. 
- Provites interface for indicator data in the widest meaning of the word. 

- Has one Master and Worker modus where the proces becomes scaleble 
- Has one Swift modus, that makes is possible to run on LOW-CPU single container. 
## Class structure

![Stock Data classstrucrtre ](https://www.trend-impact-service.nl/static/StockDataClasses.png)

## Database erd v1

![Stock Data Orm ](https://www.trend-impact-service.nl/static/StockDataService.png)

## test the application: 
if you need to test the repository's and main services. 
Move to the tests folder
```
cd src/StockDataService/Tests
```
then run the Unittest with. 
```
python -m unittest discover
```

## To get production started: 
**Note: For Production**

# Master/Worker(Scalible) modus: 
its very important to understand that the redis container needs to run with the abilty to extend. run ```sysctl vm.overcommit_memory=1``` to make sure that this is possible. 


**Enable the target machine**: 
1. Download the official docker : https://docs.docker.com/engine/install/ubuntu/
2. Install and start a redis service. : ```apt-get install redis``` && ```redis-server --daemonize yes```. If you want to check if redis runs: ```ps aux | grep redis-server```. If you want to see the input that is written to redis :  ```redis-cli monitor``` 

1. pull the repo and fill in the .env file in the format below. 
2. if its needed, make sure that all tasks a purged, move to the main folder and run:
```
celery -A StockDataService.Process.task_manager.task_schedular purge
```
3. Make sure there is a .env file in the directory matching the template.  
4. Startup the container:
```
docker-compose -f docker-compose.production.yml --env-file .env up --build
``` 

# To start the swift (low CPU)
1. Pull the repo and fill in the .env file
2. Run
```
docker-compose -f docker-compose.yml --env-file .env up --build
``` 
3.  potentially you can check in the database-log to check if startup correcly. 


## Getting Started development (Windows)

To get started with the Stock Data Service, follow these steps:

1. pull the repo.
2. Create a venv with
```
python -m venv venv && .\venv\Scripts\activate

```
3. Setup envoirement, this will install the library's and setup the package.

```
pip install -e . 
```
4. To find the python interpeter where the package is installed: 
```
where python
```
# if you would like to continue to develop (Master/Worker) mode: 
5. To start the application you need to start the task-schedular function.
```
celery -A StockDataService.Process.task_manager.task_schedular beat -l info

```
6. To start the celery worker: (WINDOWS)
```
celery -A StockDataService.Process.task_manager.task_schedular worker -l info -P solo
```
on linux you run the app without -P solo. (LINUX)
```
celery -A StockDataService.Process.task_manager.task_schedular worker -l info 
```
7. If all tasks need to be reset/deleted, in any case of problem you can type the following command:
```
celery -A StockDataService.Process.task_manager.task_schedular purge
```
# if you would like to develop based on the (Swift) modus: 

5. Start the proces with
```
python src\StockDataService\Processes\update_swift.py 
```


## How to run with docker (Windows)
1. to build the docker image:
```
docker build -t stock-data-service . 
```
2. To propper start the docker, container you need to make sure there is a redis DB on the standard port. Then run thee command: 
```
docker run -d --restart always --network host stock-data-service

```
3. To run the full application with docker and redis you can run: 

if you want to test the container: 

```
docker build -t stock-data-service .
docker run -it stock-data-service

```

## Getting Started development (Linux)
Step 1. 
pull the repo: 
```
git pull https://github.com/HanhartTradingTechnologies/StockDataService/
Username : login
Password : your github token

```
2. Create a venv with
```
python3 -m venv venv && source venv/bin/activate
curl https://bootstrap.pypa.io/get-pip.py | python
deactivate
source venv/bin/activate

```
3. Setup envoirement, this will install the library's and setup the package.

```
pip install -e . 
```
4. Set permissions

```
chmod +x ./startup.sh
```

## Setup the .env file
Make sure there is always a .env file attached in the app so it can run without problems:
This is template for the .env file: 

**important** there are 3 system modus that are set in STATUS:
MAIN = Task scedular core
WORKER = A task processer 
SWIFT = an single core application with low CPU usage. 
***this is done in the DockerCompose files but JIC you could add it too***
# .env file
```
# PostgreSQL Configuration
POSTGRES_HOST=****
POSTGRES_PORT=****
POSTGRES_DB=****
POSTGRES_USER=****
POSTGRES_PASSWORD=****


# Redis Configuration
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

# Redis Configuration
ZEROMQ_PORT=5555



```

## Setup pipeline 
if you have Github runner installed, continues deployment is possible. 
Make sure the user that is selected in action runner has all permissions in the workfolder in the action_runner.