prophecy-orchestrate 1.0.19.dev61
Prophecy Orchestrate - High-performance data processing library with Go backend
Requires Python: >=3.8

# Project Setup Guide

## Prerequisites

### 1. Install Go
```bash
brew install go
```

### 2. Configure Go Environment
Add these lines to `.bash_profile` or `.zshrc`:
```bash
export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$PATH
```
> Note: After adding these lines, restart your terminal or run `source .zshrc`

### 3. Create Go Directory Structure
```bash
mkdir -p $GOPATH/{bin,pkg,src}
```

### 4. Install Dependencies
1. Install Protobuf:
   ```bash
   brew install protobuf
   ```

2. Install cmake
   ```bash
   brew install cmake
   ```

2. Install gRPC:
   ```bash
   go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1
   ```

4. Install Project Dependencies:
   ```bash
   make install-dependencies 
   ```
5. init/update all submodules
   ```bash
   git submodule update --init --recursive
   ```

## Development

### Building and Testing
- Build and run the project:
  ```bash
  make all
  ```

- Run unit tests:
  ```bash
  make test
  ```

### Running the Server Locally

#### 1. Run Orchestrator
```bash
export RESOURCEMANAGER_TYPE=local
export LOG_FORMAT=console
go run ./pkg/cmd
```

#### 2. Run Task Coordinator
```bash
export RESOURCEMANAGER_TYPE=local
export LOG_FORMAT=console
export MODE=task_coordinator
export DUMPTRACEONEXIT=true
export PPROFADDR=":6061"
go run ./pkg/cmd
```

#### Logging Options
- For JSON logging: `unset LOG_FORMAT`
- For trace logging: Add `-trace` flag
- For debug logging: Add `-debug` flag

### Running with Docker
1. Build Docker images:
   ```bash
   make docker-build
   ```

2. Run with docker-compose:
   ```bash
   ./run_docker_compose_test.sh
   ```

## Monitoring and Administration

### Admin UI
- View Pipelines, Runs & Details: [http://localhost:9089/engine/view](http://localhost:9089/engine/view)
- Debug Profiler: [http://localhost:6060/debug/pprof](http://localhost:6060/debug/pprof)

### Debugging
```shell
go tool pprof profile
(pprof)> tree 
(pprof)> web 
```

### Runtime Configuration
Set log level dynamically:
```bash
curl --location 'http://localhost:9089/log/level' \
  --header 'Content-Type: application/json' \
  --data '{
    "level": "debug"
  }' 
```

## Gem Debugging

### Snowflake

#### CLI
- export table from snowflake
```sql
COPY INTO '@%TUSHAR/prophecy/data1row' from tushar file_format = (type = parquet) OVERWRITE = TRUE;

```
- download the parquet file in local
```shell
pip install snowflake-cli-labs
snow sql --query 'GET @%TUSHAR/prophecy/data file:///Users/tushargarg/;'
```

# Prophecy DBT

A lightweight, stateful DBT parser and template resolver that leverages dbt-core libraries while providing a simpler API.

## Overview

Prophecy DBT is designed to create a lighter, stateful version of DBT that can be used for template resolution, parsing, and executing SQL commands with materialization. It leverages dbt-core's libraries when available, providing the best of both worlds:

1. Compatibility with core DBT functionality
2. Lightweight processing when full DBT functionality isn't needed
3. Stateful handling of models, macros, and sources
4. Simple API for integration with web services and applications

## Features

- **Optimized Jinja Resolution**: Uses dbt-core's built-in Jinja rendering when available, with fallback to a lightweight implementation
- **Stateful Entity Management**: Tracks models, macros, and sources in memory for efficient updates
- **Multiple Access Methods**: CLI, REST API, and GRPC services for integration
- **Smart Materialization**: Generates proper SQL for table/view creation based on model configuration
- **High Performance**: Designed for speed and efficiency when processing templates
- **Automatic DBT Context**: Injects appropriate context variables based on DBT project structure

## Installation

```bash
# Basic installation
pip install prophecy_dbt
```

## Start Server

```bash
export LOG_FORMAT=console && export ENABLE_PRODUCTION_DEBUGGING=true && export DBT_DBT_EAGER_LOAD_PROFILE=false && python -m server --log-level DEBUG
```

## Usage

### Command Line

```bash
# Resolve a simple template
prophecy-dbt resolve-template "SELECT * FROM {{ ref('model_name') }}" --project-dir ./my_dbt_project

# Resolve a file with materialization
prophecy-dbt resolve-file ./models/my_model.sql --materialization --project-dir ./my_dbt_project
```

### Python API

```python
from parser.jinja_resolve import resolve_template, resolve_with_materialization

# Simple template resolution
sql = resolve_template(
    template="SELECT * FROM {{ ref('model_name') }}",
    project_dir="./my_dbt_project"
)
```

### REST API

Make a request for Materiazliation:

```bash
curl --location 'http://localhost:50054/resolve_jinja_templates' \
--header 'Content-Type: application/json' \
--data '{
    "entity":"orders",
    "mode": "RUN",
    "entity_type": "MODEL",
    "parameters": {
        "test_amount": "200"
    }
}'
```

Make a request for compiled DBT Code:

```bash
curl --location 'http://localhost:50054/resolve_jinja_templates' \
--header 'Content-Type: application/json' \
--data '{
    "entity":"orders",
    "mode": "COMPILE",
    "entity_type": "MODEL",
    "parameters": {
        "test_amount": "200"
    }
}'
```

## Architecture

The package is designed with a layered architecture:

1. **Core Layer**: Basic Jinja resolution without DBT dependencies
2. **DBT Integration Layer**: Enhanced resolution using dbt-core libraries
3. **Stateful Layer**: Adds in-memory tracking of entities and dependencies
4. **API Layer**: Provides various interfaces (CLI, REST, GRPC)

This design allows the package to work with or without dbt-core installed, adapting to the available dependencies.

## How It Works

The package includes a smart integration with dbt-core that leverages the following components:

1. **dbt.clients.jinja**: For optimal Jinja rendering
2. **dbt.context**: For providing the proper variables and functions
3. **dbt.parser**: For analyzing SQL and extracting metadata

When dbt-core is not available, it falls back to a custom implementation that covers the essential functionality.