formenergy-observability 0.3.2
OTel tracing/logging tools, emphasizing data pipeline use cases with Dagster.
Requires Python: >=3.8

# Form Energy Observability Python Library

OTel tracing and logging tools, emphasizing data pipeline use cases with Dagster.

## Using the Library

### [Install from PyPI](https://pypi.org/project/formenergy-observability/)

`pip install formenergy-observability`

### Features

See the [examples/](examples/) directory for examples you can run and screenshots of the traces they produce.

Some key features demonstrated in the examples:

Context-aware tracing, so that spans and events can inherit attributes from parents:

```py
from form_observability import ContextAwareTracer, ctx

_trace = ContextAwareTracer(__name__)

with ctx.set({"my_attribute_name": my_attribute_value}):
    # There could be many intervening function calls and spans.
    # Spans and events in this context automatically have the my_attribute_name
    # attribute set on them.
    _trace.add_event("my_event")
```

Trace context propagation for Dagster ops:

```py
from form_observability.dagster_otel import otel_op, publish_current_trace_context
from form_observability import ContextAwareTracer

_trace = ContextAwareTracer(__name__)

@op
def root_trace_context_op(context):
    with _trace.start_as_current_span("root"):
        # writes the trace context (as an asset materialization event) for later ops
        publish_current_trace_context(context)

@otel_op()  # picks up the trace context and starts a new span for this op
def my_op(context):
    _trace.add_event("my_event")

@graph
def my_graph()
    # In the graph, publish a root trace using a regular op, and then subsequent
    # @otel_op ops will pick it up and contribute to the same trace.
    setup_done = root_trace_context_op()
    my_op(start=setup_done)
```

### Running the Examples

Get an API key for uploading traces to your observability service (preferably in a development environment), set it in your environment, and run the examples. This should publish traces that you can then view.

```bash
export OTEL_EXPORTER_OTLP_HEADERS_JSON="{'x-honeycomb-team': '<API key>'}"
export OTEL_EXPORTER_OTLP_ENDPOINT="your observability service, defaults to Honeycomb"
python examples/basics.py
dagster job execute --python-file examples/dagster_job.py
```

## License

This library is provided under the MIT license, see [LICENSE](LICENSE).

## Release Notes

*   0.3.2 Update `grpcio` minimum dependency version to 1.48.1 and `dagster` to 1.5.0. Thanks @dang-kaiko !
*   0.3.1 Fix type used for `otlp_headers` to be a `dict`. Thanks @spenczar !
*   0.3.0 Add `ContextAwareTracer.start_new_linked_trace`.
*   0.2.0 Switch to `context.op_handle` and `materialization.metadata` as dict for Dagster 1.2.
*   0.1.0 Initial external release, including `ContextAwareTracer`, `ctx`, and `@otel_op`.

## Development

If you contribute, please ensure you and your employer accept [the license](LICENSE) for any of your contributions.

### Setup

Create and activate a virtual environment with your favorite tool, such as [direnv](https://github.com/direnv/direnv/wiki/Python). This project requires Python 3.8.

```bash
pip install flit
flit install --deps all --symlink [--python /path/to/venv/python]
```

By default, `flit` installs into the current virtual environment. Using the `--python` argument can let you install into a specific venv other than the currently active one. For example you can activate some other venv, run `which python`, and then pass the output of that command to `flit`'s `--python` argument when installing from the `form-observability` repo.

Initialize pre-commit hooks:

```bash
pre-commit install --hook-type pre-commit --hook-type pre-push
```

### Running Tests

From the project root directory:

```bash
pytest
```