semvec/pypi-packages huggingface dataset:
nodestream-plugin-meta 0.15.0a0
A plugin to nodestream for building a graph of the schema of the graph.
License: Apache 2.0
Requires Python: <4.0,>=3.10
Keywords: etl, neo4j, declarative, data, kafka, ingest, nodestream

# Nodestream Meta Plugin

This plugin allows you to build a Graph of your Graph's Schema.

## Installation

```bash
pip install nodestream-plugin-meta
```

## Meta-Schema

The schema of the graph schema is as follows:

![Meta Schema](meta.png)

For extra meta characteristics, the schema of the schema schema is also added to the projects schema like any other pipeline.
In other words, the types `NodeType`, `Property`, and so on are added to the schema of the graph. 

## Getting Started

Unlike other schema plugins, this plugin uses the ingestion system to build the metagraph in your graph. 
That means that you need to build a pipeline to do the ingestion.
For the simple case, you can build a single step pipeline that looks like this:

```yaml
- implementation: nodestream_plugin_meta:SchemaRenderer
```

This will locate the project file located inside the current working directory (or its parent directories) and 
render the schema to the graph. You can also specify the path to the project file:

```yaml
- implementation: nodestream_plugin_meta:SchemaRenderer
  parameters:
    project_path: /path/to/nodestream.yaml
```

You may also specify an overrides file to enrich the inferred schema:

```yaml
- implementation: nodestream_plugin_meta:SchemaRenderer
  parameters:
    project_path: /path/to/nodestream.yaml
    overrides_path: /path/to/overrides.yaml
```

The overrides file should look like this:

```yaml
nodes:
  - name: Person
    properties:
      is_cool:
        type: BOOLEAN
relations:
  - name: Knows
    properties:
      since:
        type: DATE
```

The file is merged with the inferred schema, so you can add new nodes, relations, or properties, or override parts of existing ones.

## Querying the Meta Graph

The meta graph can be queried like any other graph data. 
For example, to get all the node types in the graph:

```cypher
MATCH (n:NodeType) RETURN n.name
```

Or to get all the properties of a node type:

```cypher
MATCH (n:NodeType {name: 'Person'})-[:HAS_PROPERTY]->(p:Property) RETURN p.name, p.type, p.is_key
```

Same goes for relations and their properties:

```cypher
MATCH (r:RelationshipType {name: 'Knows'})-[:HAS_PROPERTY]->(p:Property) RETURN p.name, p.type, p.is_key
```

Also you can get the adjacencies the graph schema:

```cypher
MATCH (a:NodeType)<-[:TO|FROM]-(adj:Adjacency)-[:TO|FROM]->(b:NodeType)
MATCH (adj)-[:THROUGH]->(r:RelationshipType)
RETURN a.name, r.name, b.name
```
nodestream-plugin-k8s 0.15.0a0
A plugin for nodestream that manages k8s resources.
License: Apache 2.0
Requires Python: <4.0,>=3.10

# Nodestream Kubernetes Plugin 

NOTE: This plugin is still in development and is not ready for production use.

## Overview

This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, `nodstream-plugin-k8s` will be able to manage your kubernetes resources for you. 

## Installation

```bash
pip install nodestream-plugin-k8s
```

## Usage

### 1. Create a `nodestream.yaml` file and project.

```yaml
# nodestream.yaml
scopes:
   crons: 
      pipelines:
        - path: pipelines/crons/my_scheduled_pipeline.yaml
  perpetual:
      pipelines:
        - path: pipelines/perpetual/my_kafka_pipeline.yaml
```

### 2. Annotate your pipeline with `nodestream-plugin-k8s` annotations.

```yaml
# nodestream.yaml
scopes:
   crons:
      pipelines:
        - path: pipelines/crons/my_scheduled_pipeline.yaml
          annotations:
            nodestream_plugin_k8s_schedule: "0 0 * * *"
    perpetual:
       pipelines:
          - path: pipelines/perpetual/my_kafka_pipeline.yaml
            annotations:
              nodestream_plugin_k8s_conccurency: 1
```

### 3. Run `nodestream-plugin-k8s` to deploy your pipelines.

```bash
nodestream k8s sync --namespace my-nodestream-app-namespace 
```


### 4. Check your kubernetes cluster for your deployed pipelines.

```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```

### 5. Update your `nodestream.yaml` file and re-run `nodestream-plugin-k8s` to update your pipelines.

```yaml
# nodestream.yaml
scopes:
   crons:
      pipelines:
        - path: pipelines/crons/my_scheduled_pipeline.yaml
          annotations:
            nodestream_plugin_k8s_schedule: "30 0 * * *"
    perpetual:
       pipelines:
          - path: pipelines/perpetual/my_kafka_pipeline.yaml
            annotations:
              nodestream_plugin_k8s_conccurency: 100
```

```bash
nodestream k8s sync --namespace my-nodestream-app-namespace 
```

### 6. Check your kubernetes cluster for your updated pipelines.

```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```

### 7. Delete your pipelines.

```bash
nodestream k8s destroy --namespace my-nodestream-app-namespace 
```

## Annotations

### `nodestream_plugin_k8s_schedule`

This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression. 

### `nodestream_plugin_k8s_concurrency`

This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.

### `nodestream_plugin_k8s_image`

This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.