semvec/pypi-packages huggingface dataset:
dagster-weaviate 0.0.3
Dagster integration with Weaviate
Requires Python: >=3.10

# dagster-weaviate

A dagster module that provides integration with [Weaviate](https://weaviate.io/)
(both Cloud and Self-Hosted Weaviate instances).

## Installation

The `dagster_weaviate` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-weaviate
```

## Example Usage (Local Weaviate Instance)

(Based on the [Weaviate Quickstart Guide (Local)](https://weaviate.io/developers/weaviate/quickstart/local))

```python
from dagster import Definitions, asset
from dagster_weaviate import WeaviateResource, LocalConfig

@asset
def my_table(weaviate: WeaviateResource):
    with weaviate.get_client() as weaviate_client:
        questions = weaviate_client.collections.get("Question")
        questions.query.near_text(query="biology", limit=2)

defs = Definitions(
    assets=[my_table],
    resources={
        "weaviate": WeaviateResource(
            connection_config=LocalConfig(
                host="192.168.0.10",
                port=8080,
            )
        ),
    },
)
```


## Example Usage (Weaviate Cloud Instance)

Based on the [Weaviate Cloud Quickstart Guide](https://weaviate.io/developers/wcs/quickstart)

```python
from dagster import Definitions, asset
from dagster_weaviate import WeaviateResource, CloudConfig

@asset
def my_table(weaviate: WeaviateResource):
    with weaviate.get_client() as weaviate_client:
        questions = weaviate_client.collections.get("Question")
        questions.query.near_text(query="biology", limit=2)

defs = Definitions(
    assets=[my_table],
    resources={
        "weaviate": WeaviateResource(
            connection_config=CloudConfig(
                cluster_url=wcd_url
            ),
            auth_credentials={
                "api_key": wcd_apikey
            },
            headers={
                "X-Cohere-Api-Key": cohere_apikey,
            }
        ),
    },
)
```


## Development

The `Makefile` provides the tools required to test and lint your local installation

```sh
make test
make ruff
make check
```
dagster-qdrant 0.0.3
Dagster integration with Qdrant
Requires Python: >=3.10

# dagster-qdrant

A Dagster module that provides an integration with [Qdrant](https://qdrant.tech/).

## Installation

The `dagster_qdrant` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-qdrant
```

## Example Usage

You can get a free-forever cloud instance from [cloud.qdrant.io](http://cloud.qdrant.io).

```python
from dagster_qdrant import QdrantConfig, QdrantResource

import dagster as dg


@dg.asset
def my_table(qdrant_resource: QdrantResource):
    with qdrant_resource.get_client() as qdrant:
            qdrant.add(
                collection_name="test_collection",
                documents=[
                    "This is a document about oranges",
                    "This is a document about pineapples",
                    "This is a document about strawberries",
                    "This is a document about cucumbers",
                ],
            )
            results = qdrant.query(
                collection_name="test_collection", query_text="hawaii", limit=3
            )


defs = dg.Definitions(
    assets=[my_table],
    resources={
        "qdrant_resource": QdrantResource(
            config=QdrantConfig(
                host="xyz-example.eu-central.aws.cloud.qdrant.io",
                api_key="<your-api-key>",
            )
        )
    },
)

```

## Development

The `Makefile` provides the tools required to test and lint your local installation.

```sh
make test
make ruff
make check
```
dagster-notdiamond 0.0.4
Dagster integration with Not Diamond
Requires Python: <3.13,>=3.10

# Dagster resource for Not Diamond ¬◇

Not Diamond is an AI model router that automatically determines which LLM is best-suited to respond to any query, improving LLM output quality by combining multiple LLMs into a **meta-model** that learns when to call each LLM.

## Installation

```bash
pip install dagster-notdiamond
```

## Configuration

Make sure you have followed the steps in the [Quickstart docs][quickstart], and added a Not Diamond API key to your environment.

## Usage

Define a resource for Not Diamond, then create the asset:

```python
from dagster import (
    Definitions,
    EnvVar,
    ScheduleDefinition,
    AssetSelection,
    AssetExecutionContext,
    MaterializeResult,
    asset,
)
from dagster_notdiamond import NotDiamondResource


@asset
def notdiamond_asset(context: AssetExecutionContext, notdiamond: NotDiamondResource) -> MaterializeResult:
    with notdiamond.get_client(context) as client:
        session_id, best_llm = client.model_select(
            model=["openai/gpt-4o", "openai/gpt-4o-mini"],
            messages=[{"role": "user", "content": "Say this is a test"}]
        )
    return MaterializeResult(metadata={"session_id": session_id, "best_llm": str(best_llm)})

notdiamond_schedule = ScheduleDefinition(
    name="notdiamond_schedule",
    target=AssetSelection.all(),
    cron_schedule="17 * * * *",
)

defs = Definitions(
    assets=[notdiamond_asset],
    resources={
        "notdiamond": NotDiamondResource(api_key=EnvVar("NOTDIAMOND_API_KEY")),
    },
    schedules=[notdiamond_schedule],
)
```

Please refer to `examples/example_notdiamond.py` for an example script which you can test by updating
your pyproject.toml as follows (replacing `$PROJECT_HOME` with the path to this project):

```toml
[tool.dagster]
module_name = "examples.example_notdiamond"
working_directory = "$PROJECT_HOME/libraries/dagster-notdiamond/"
```

## Support

Visit the [documentation for Not Diamond][docs] or send a message to [support@notdiamond.ai][mailto]

[docs]: https://docs.notdiamond.ai
[mailto]: mailto:support@notdiamond.ai
[quickstart]: https://docs.notdiamond.ai/docs/quickstart
dagster-modal 0.0.4
Dagster integration with Modal
Requires Python: >=3.10

# dagster-modal

Leverage the scalable compute platform of Modal from your Dagster pipelines.

## Setup

```sh
uv venv create
source .venv/bin/activate
```

```sh
make install
```

## Test

```sh
make test
```

## Build

```sh
make build
```
dagster-hex 0.1.9
Dagster integration with Hex
Requires Python: >=3.10

# Dagster Hex Integration

### Installation

To install the library, use pip alongside your existing Dagster environment.

```bash
pip install dagster-hex
```

### Usage

Create a `HexResource` specifying the `api_key`, then, use that resource in our assets.

```python
import os
import dagster as dg

from dagster_hex.resources import HexResource


@dg.asset
def custom_hex_asset(
    context: dg.AssetExecutionContext,
    hex: HexResource
) -> dg.MaterializeResult:
    project_id = "abcdefgh-1234-abcd-1234-abcdefghijkl"
    context.log.info(f"Running Hex project {project_id}")
    response = hex.run_project(
        project_id,
        inputs={
            "message": "Hello, World!"
        }
    )
    return dg.MaterializeResult(
        metadata={
            "data": response.get("data")
        }
    )


defs = Definitions(
    assets=[custom_hex_asset],
    resources={
        "hex": HexResource(
            api_key=dg.EnvVar("HEX_API_KEY")
        )
    }
)
```

#### Legacy

##### Ops

The `hex_project_op` will call the Hex API to run a project until it completes.

```python
import os
import dagster as dg

from dagster_hex.ops import hex_project_op
from dagster_hex.resources import hex_resource


hex_resource = HexResource(
    api_key=dg.EnvVar("HEX_API_KEY")
)

run_hex_op = hex_project_op.configured({
    "name": "run_hex_project_op",
    "project_id": "abcdefgh-1234-abcd-1234-abcdefghijkl"
})

@job(resource_defs={"hex": my_resource})
def hex_job():
    run_hex_op()
```

##### Asset Materializations

Ops will return an `AssetMaterialization`  with the following keys:

```
run_url
run_status_url
trace_id
run_id
elapsed_time
notifications
```
dagster-gemini 0.0.4
Dagster integration with Gemini
Requires Python: >=3.10

# dagster-gemini

A dagster module that provides integration with [Gemini](https://deepmind.google/technologies/gemini/).

## Installation

The `dagster_gemini` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-gemini
```

## Example Usage

In addition to wrapping the Gemini GenerativeModel class (get_model/get_model_for_asset methods),
this resource logs the usage of the Gemini API to to the asset metadata (both number of calls, and tokens).
This is achieved by wrapping the GenerativeModel.generate_content method.

Note that the usage will only be logged to the asset metadata from an Asset context -
not from an Op context.
Also note that only the synchronous API usage metadata will be automatically logged -
not the streaming or batching API.

```python
from dagster import AssetExecutionContext, Definitions, EnvVar, asset, define_asset_job
from dagster_gemini import GeminiResource


@asset(compute_kind="gemini")
def gemini_asset(context: AssetExecutionContext, gemini: GeminiResource):
    with gemini.get_model(context) as model:
        response = model.generate_content(
            "Generate a short sentence on tests"
        )

defs = Definitions(
    assets=[gemini_asset],
    resources={
        "gemini": GeminiResource(
            api_key=EnvVar("GEMINI_API_KEY"),
            generative_model_name="gemini-1.5-flash"
        ),
    },
)
```


## Development

The `Makefile` provides the tools required to test and lint your local installation

```sh
make test
make ruff
make check
```
dagster-ducklake 0.0.2
Dagster integration for Ducklake
Requires Python: >=3.10

# dagster-ducklake

A dagster module that provides integration with [ducklake](https://ducklake.select/)

## Installation

The `dagster_ducklake` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-ducklake
```

## Example Usage

```python
import dagster as dg

@dg.asset
def my_ducklake_asset(ddb: DuckDBConnectionProvider):
    with ddb.duckdb_connect() as con:
        query = "select * from table"

        df = con.query(query).df()
        df.head()
```

## Resource initialization

```python
{
    "ducklake": DuckLakeResource(
        metadata_backend=PostgresConfig(
            host="db.mycorp.com",
            port=5432,
            database="ducklake_catalog",
            user=dg.EnvVar("POSTGRES_USER"),
            password=dg.EnvVar("POSTGRES_PASSWORD"),
        ),
        storage_backend=S3Config(
            endpoint_url="objectstore.mycorp.com",
            bucket="duckpond-dev",
            prefix="stage",
            aws_access_key_id=dg.EnvVar("OBJECT_STORE_USER"),
            aws_secret_access_key=dg.EnvVar("OBJECT_STORE_PASSWORD"),
            region=dg.EnvVar("OBJECT_STORE_REGION"),
            use_ssl=True,
            url_style="path",
        ),
        alias="stage",
        plugins=["postgres", "httpfs", "ducklake"],
    ),
}

```

## Development

The `Makefile` provides the tools required to test and lint your local installation

```sh
make test
make ruff
make check
```
dagster-chroma 0.0.4
Dagster integration with chroma
Requires Python: >=3.10

# dagster-chroma

A dagster module that provides integration with [chroma](https://www.trychroma.com/).
Both HTTP connections, and local (filesystem-based) connections are supported.

## Installation

The `dagster_chroma` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-chroma
```

## Example Usage (Local Chroma Instance)

(Based on the [Chroma getting started guide](https://docs.trychroma.com/docs/overview/getting-started))

```python
import os
from dagster import Definitions, asset
from dagster_chroma import ChromaResource, LocalConfig, HttpConfig

@asset
def my_table(chroma: ChromaResource):
    with chroma.get_client() as chroma_client:
        collection = chroma_client.create_collection("fruits")

        collection.add(
            documents=[
                "This is a document about oranges",
                "This is a document about pineapples",
                "This is a document about strawberries",
                "This is a document about cucumbers"],
            ids=["oranges", "pineapples", "strawberries", "cucumbers"],
        )

        results = collection.query(
            query_texts=["hawaii"],
            n_results=1,
        )

defs = Definitions(
    assets=[my_table],
    resources={
        "chroma": ChromaResource(
            connection_config=
                LocalConfig(persistence_path="./chroma") if os.getenv("DEV") else
                    HttpConfig(host="192.168.0.10", port=8000)
        ),
    }
)
```

## Development

The `Makefile` provides the tools required to test and lint your local installation.

```sh
make test
make ruff
make check
```
dagster-anthropic 0.0.5
Dagster integration with Anthropic
Requires Python: >=3.10

# dagster-anthropic

A dagster module that provides integration with [Anthropic](https://www.anthropic.com//).

## Installation

The `dagster_anthropic` module is available as a PyPI package - install with your preferred python
environment manager (We recommend [uv](https://github.com/astral-sh/uv)).

```
source .venv/bin/activate
uv pip install dagster-anthropic
```

## Example Usage

In addition to wrapping the Anthropic client (get_client/get_client_for_asset methods),
this resource logs the usage of the Anthropic API to to the asset metadata (both number of calls, and tokens).
This is achieved by wrapping the Anthropic.messages.create method.

Note that the usage will only be logged to the asset metadata from an Asset context -
not from an Op context.
Also note that only the synchronous API usage metadata will be automatically logged -
not the streaming or batching API.

```python
from dagster import AssetExecutionContext, Definitions, EnvVar, asset, define_asset_job
from dagster_anthropic import AnthropicResource


@asset(compute_kind="anthropic")
def anthropic_asset(context: AssetExecutionContext, anthropic: AnthropicResource):
    with anthropic.get_client(context) as client:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[
                {"role": "user", "content": "Say this is a test"}
            ]
        )

defs = Definitions(
    assets=[anthropic_asset],
    resources={
        "anthropic": AnthropicResource(api_key=EnvVar("ANTHROPIC_API_KEY")),
    },
)
```


## Development

The `Makefile` provides the tools required to test and lint your local installation

```sh
make test
make ruff
make check
```