opentelemetry-exporter-cloudwatch-emf 0.2.0
OpenTelemetry metrics exporter for AWS CloudWatch using Embedded Metric Format (EMF)
License: Apache-2.0
Requires Python: >=3.9
Keywords: opentelemetry, cloudwatch, emf, metrics, aws, observability

# OpenTelemetry CloudWatch EMF Exporter

[![PyPI version](https://badge.fury.io/py/opentelemetry-exporter-cloudwatch-emf.svg)](https://badge.fury.io/py/opentelemetry-exporter-cloudwatch-emf)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

An OpenTelemetry metrics exporter for AWS CloudWatch using [Embedded Metric Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html).

## Why?

OpenTelemetry doesn't have a native Python SDK for exporting metrics directly to CloudWatch. The recommended approach is to run an ADOT Collector sidecar, which isn't always practical (e.g., AWS App Runner, Lambda, simple deployments).

This exporter prints EMF-formatted JSON to stdout. When running in AWS environments, CloudWatch Logs automatically parses EMF and creates CloudWatch Metrics—no collector required.

## Installation

```bash
pip install opentelemetry-exporter-cloudwatch-emf
```

## Quick Start

```python
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry_exporter_cloudwatch_emf import CloudWatchEMFExporter

# Create exporter
exporter = CloudWatchEMFExporter(namespace="MyApplication")

# Set up meter provider
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=60000)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

# Create metrics
meter = metrics.get_meter("my-service")
counter = meter.create_counter("requests", unit="1", description="Request count")
histogram = meter.create_histogram("latency", unit="ms", description="Request latency")

# Record metrics
counter.add(1, {"endpoint": "/api/users", "method": "GET"})
histogram.record(45.2, {"endpoint": "/api/users"})
```

## Output

The exporter prints EMF JSON to stdout:

```json
{"_aws":{"Timestamp":1700000000000,"CloudWatchMetrics":[{"Namespace":"MyApplication","Dimensions":[["endpoint","method"]],"Metrics":[{"Name":"requests","Unit":"Count","StorageResolution":60}]}]},"requests":1,"endpoint":"/api/users","method":"GET"}
```

CloudWatch Logs automatically parses this and creates metrics in the `MyApplication` namespace.

## Configuration

```python
exporter = CloudWatchEMFExporter(
    namespace="MyApp",              # Required: CloudWatch namespace (1-256 chars)
    output=sys.stdout,              # Output stream (default: stdout)
    dimension_keys=["service"],     # Filter dimensions (default: all attributes)
    max_dimensions=10,              # Max dimensions per metric (default: 30)
    storage_resolution=1,           # 1=high-res, 60=standard (default: 60)
    log_group_name="/aws/my-app",   # Optional: CloudWatch Log Group for routing
    log_stream_name="stream-1",     # Optional: CloudWatch Log Stream for routing
    auto_detect_aws=True,           # Auto-detect Lambda/ECS environment (default: True)
    histogram_as_values=True,       # Emit histograms as Values arrays (default: True)
)
```

### Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `namespace` | str | Required | CloudWatch metrics namespace |
| `output` | TextIO | stdout | Output stream for EMF JSON |
| `dimension_keys` | list | None | Attribute keys to use as dimensions (None = all) |
| `max_dimensions` | int | 30 | Maximum dimensions per metric (CloudWatch limit) |
| `storage_resolution` | int | 60 | 1 for high-resolution, 60 for standard |
| `timestamp_fn` | callable | None | Custom timestamp function (returns epoch ms) |
| `log_group_name` | str | None | CloudWatch Log Group name for EMF routing |
| `log_stream_name` | str | None | CloudWatch Log Stream name for EMF routing |
| `auto_detect_aws` | bool | True | Auto-detect AWS environment (Lambda, ECS) |
| `histogram_as_values` | bool | True | Emit histograms as Values arrays for percentiles |

## Supported Metric Types

| OTel Type | EMF Output |
|-----------|------------|
| Counter | Single metric value |
| Gauge | Single metric value |
| Histogram | Values array (default) or `{name}_count/_sum/_min/_max` |
| ExponentialHistogram | Values array (default) or derived metrics |

### Histogram Handling

By default (`histogram_as_values=True`), histograms are exported as Values arrays, which enables **percentile calculations** (p50, p99, etc.) in CloudWatch.

If you set `histogram_as_values=False`, histograms are converted to four derived metrics:

| Derived Metric | Description | Unit |
|---------------|-------------|------|
| `{name}_count` | Number of observations | Count |
| `{name}_sum` | Sum of all observed values | Original unit |
| `{name}_min` | Minimum observed value | Original unit |
| `{name}_max` | Maximum observed value | Original unit |

## AWS Environment Auto-Detection

When `auto_detect_aws=True` (default), the exporter automatically detects and adds dimensions for:

**AWS Lambda:**
- `aws_lambda_function_name`
- `aws_lambda_function_version`
- `aws_region`
- `aws_lambda_memory`

**Amazon ECS/Fargate:**
- `aws_ecs`
- `aws_region`

## Log Group/Stream Routing

For non-Lambda environments (EC2, containers), you can specify log group and stream names in the EMF output. The CloudWatch Agent or Fluent Bit will use these to route metrics:

```python
exporter = CloudWatchEMFExporter(
    namespace="MyApp",
    log_group_name="/aws/ec2/my-app",
    log_stream_name="i-1234567890abcdef0",
)
```

## Unit Mapping

OpenTelemetry units (UCUM) are automatically mapped to CloudWatch units:

| OTel | CloudWatch |
|------|------------|
| `s` | Seconds |
| `ms` | Milliseconds |
| `By` | Bytes |
| `KiBy` | Kilobytes |
| `1` | Count |
| `%` | Percent |

## AWS Environment

EMF works automatically in these AWS environments:
- **Lambda**: CloudWatch Logs integration built-in
- **ECS/Fargate**: Logs sent to CloudWatch via awslogs driver
- **App Runner**: Stdout automatically sent to CloudWatch Logs
- **EC2**: Use CloudWatch agent to send logs

## Best Practices

### Dimension Cardinality

Be careful with high-cardinality dimensions (user IDs, request IDs). Each unique dimension combination creates a separate metric, impacting costs.

```python
# Good: Low cardinality
exporter = CloudWatchEMFExporter(
    namespace="MyApp",
    dimension_keys=["service", "environment", "endpoint"],
)

# Bad: High cardinality (avoid)
# dimension_keys=["user_id", "request_id"]
```

### High-Resolution Metrics

Use `storage_resolution=1` for sub-minute granularity (additional cost):

```python
exporter = CloudWatchEMFExporter(
    namespace="MyApp",
    storage_resolution=1,  # 1-second resolution
)
```

### Percentile Support

For percentile statistics (p50, p99), keep `histogram_as_values=True` (default):

```python
# Enables percentile calculations in CloudWatch
exporter = CloudWatchEMFExporter(
    namespace="MyApp",
    histogram_as_values=True,  # Default
)
```

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black src tests
isort src tests

# Lint
flake8 src tests
mypy src
```

## License

Apache 2.0 - See [LICENSE](LICENSE) for details.

## Contributing

Contributions welcome! Please open an issue or PR.

## Related

- [OpenTelemetry Python](https://github.com/open-telemetry/opentelemetry-python)
- [CloudWatch EMF Specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html)
- [AWS EMF Feature Request](https://github.com/open-telemetry/opentelemetry-python-contrib/issues/2490)