lineagentic-kg 1.6.0
Turn YAML into a Knowledge Graph in minutes
Requires Python: >=3.8
Keywords: neo4j, metadata, registry, lineage, catalog

<div align="center">
  <img src="https://raw.githubusercontent.com/lineagentic/lineagentic-kg/main/images/lineagentickg.jpg" alt="Lineagentic Logo" width="880" height="500">
</div>

# Lineagentic-KG

Lineagentic-KG is a knowledge graph builder library that converts YAML definitions into a fully operational and customizable knowledge graph. While one key use case is building a data catalog, the framework is generic and extensible—making it easy to define entities, aspects, and relationships for any domain.

With automatic REST API and CLI tooling generation, Lineagentic-KG delivers a “batteries included” experience for quickly turning YAML into production-ready knowledge graph.

##Features

- **Generic Metadata Model Generator**: Define flexible, extensible models with entities, aspects, and relationships.
- **REST API Generator**: Automatically expose FastAPI endpoints from your YAML registry.
- **CLI Generator**: Instantly get CLI commands derived from the registry.
- **Type-Safe Code**: Ensure correctness and reliability in data handling.

## Quick Start

```
1. pip install lineagentic-kg
```
```python
from lineagentic_kg.registry.factory import RegistryFactory

# Initialize the registry factory with your config file
registry_factory = RegistryFactory("lineagentic_kg/config/main_registry.yaml")

# Create a Neo4j writer instance
neo4j_writer = registry_factory.create_writer(
    uri="bolt://localhost:7687",
    user="neo4j",
    password="password"
)


# Create a dataset
dataset_urn = neo4j_writer.upsert_dataset(
    platform="snowflake",
    name="customer_data",
    env="PROD"
)

# Create a corp user
user_urn = neo4j_writer.upsert_corpuser(
    username="john.doe"
)

# Create a tag
tag_urn = neo4j_writer.upsert_tag(
    key="sensitive",
    value="true"
)

# Retrieve entities
dataset = neo4j_writer.get_dataset(dataset_urn)
print(f"Retrieved dataset: {dataset}")

user = neo4j_writer.get_corpuser(user_urn)
print(f"Retrieved user: {user}")

# Clean up
neo4j_writer.close()
print("LineAgentic KG example completed successfully!")
```

## Restful API Generator:

If you want to auto-generated restful apis and save huge time for api creation you are in the right place. just run command:

```
make generate-and-run-api
```
In examples folder you can find a some example of how call api with curl, you can just run: ```./examples/api_calls_examples.sh```

## CLI Generator:

You can also auto-generate CLI tooling. Just run ``` make generate-cli ``` . Examples of cli calls can be found and run: ```./examples/cli_calls_examples.sh```


## How Lineagentic-KG Works 

Lineagentic-KG is a **dynamic code generation system** that creates knowledge graph from YAML configuration files. 

1- Registry module:Core part of the Lineagentic-KG is Registry module which is developed based on registry design pattern. Think of it as a code factory that reads configuration which is in this case is Yaml file and builds classes automatically. 

2- API-Generator: This module is responsible for generating RESTful APIs. It is developed based on FastAPI framework and leverages methods generated by Registry module and builds APIs around them.

3- CLI-Generator: This module is cli interface generator. It is developed based on Click framework and leverages methods generated by Registry module to build CLI commands.

### Why This Architecture is Powerful

This system essentially turns YAML configuration into working knowledge graph backend at runtime! It provides:

1. **Flexibility**: Change data models without code changes
2. **Consistency**: All entities follow the same patterns
3. **Maintainability**: Business logic is separated from implementation
4. **Extensibility**: Easy to add new entity types and relationships
5. **Type Safety**: Generated code ensures proper data handling


## Detailed Flow Diagrams

### 1. Bootstrap Phase: RegistryFactory Initialization
This diagram shows the complete initialization flow from YAML configuration to generated class.

<p align="center">
  <img src="images/01_bootstrap_phase.png" alt="Bootstrap Phase Diagram" width="800">
</p>
*Shows how RegistryFactory loads config, validates it, generates functions, and creates the final writer class.*

### 2. Runtime Phase: Using Generated Methods
This diagram shows what happens when you use the generated methods.

<p align="center">
  <img src="images/02_runtime_phase.png" alt="Runtime Phase Diagram" width="800">
</p>
*Flow when calling `upsert_dataset()` and `add_aspect()`.*

### 3. Configuration Loading Flow
<p align="center">
  <img src="images/03_config_loading.png" alt="Configuration Loading Diagram" width="800">
</p>

### 4. Method Generation Flow
<p align="center">
  <img src="images/04_method_generation.png" alt="Method Generation Diagram" width="800">
</p>

### 5. Overall System Architecture
<p align="center">
  <img src="images/05_system_architecture.png" alt="System Architecture Diagram" width="800">
</p>

## 6. Data Flow Overview
<p align="center">
  <img src="images/06_data_flow.png" alt="Data Flow Diagram" width="800">
</p>

## Step-by-Step Process

1. Configuration Files (`lineagentic_kg/config/` folder)

The system starts with YAML configuration files that define the data model.

- **`main_registry.yaml`**: The main entry point that includes all other config files
- **`entities.yaml`**: Defines what types of data objects exist (Dataset, DataFlow, CorpUser, etc.) 
- **`urn_patterns.yaml`**: Defines how to create unique identifiers (URNs) for each entity
- **`aspects.yaml`**: Defines properties and metadata for entities (e.g. datasetProperties, dataflowProperties, etc.)
- **`relationships.yaml`**: Defines how entities connect to each other (e.g. dataset -> dataflow)
- **`utilities.yaml`**: Defines helper functions for data processing (e.g. data cleaning, data transformation, etc.)

2. Registry Loading (`lineagentic_kg/registry/loaders.py`)

- Reads the main registry file
- Merges all included YAML files into one big configuration
- Handles file dependencies and deep merging

3. Validation (`lineagentic_kg/registry/validators.py`)

- Checks that all required sections exist
- Validates configuration structure
- Ensures everything is properly configured

4. Code Generation (`lineagentic_kg/registry/generators.py`)

- **URNGenerator**: Creates functions that generate unique identifiers
- **AspectProcessor**: Creates functions that process entity metadata
- **UtilityFunctionBuilder**: Creates helper functions for data cleaning/processing

5. Class Generation (`lineagentic_kg/registry/writers.py`)

- Takes all the generated functions and configuration
- Dynamically creates a Python class called `Neo4jMetadataWriter`
- This class has methods like:
  - `upsert_dataset()`, `get_dataset()`, `delete_dataset()`
  - `upsert_dataflow()`, `get_dataflow()`, `delete_dataflow()`
  - And so on for each entity type

6. Factory (`lineagentic_kg/registry/factory.py`)

- Orchestrates the entire process
- Creates the final writer class
- Provides a simple interface to use the generated code

## Example in fine grained way: How a Dataset Gets Created

1. **Config says**: "Dataset entities need platform, name, env, versionId properties"
2. **URN Pattern says**: "Dataset URNs should look like: `urn:li:dataset:(platform,name,env)`"
3. **Generator creates**: A function that builds URNs from the input data
4. **Writer gets**: A method `upsert_dataset(platform="mysql", name="users", env="PROD")`
5. **Result**: Creates a dataset node in Neo4j with the URN `urn:li:dataset:(mysql,users,PROD)`

## Key Benefits

- **No hardcoded entity/aspect/relationship types**: Add new entities/aspects/relationships by just editing YAML
- **Flexible URN patterns**: Change how IDs are generated without touching code
- **Dynamic methods**: New entity types automatically get create/read/delete methods
- **Configuration-driven**: Business logic is in config files, not code
- **Maintainable**: Changes to data model only require config updates