hpke-http 1.5.0
End-to-end encryption for HTTP APIs using RFC 9180 HPKE. Drop-in middleware for FastAPI, aiohttp, and httpx.
Requires Python: >=3.10
Keywords: hpke, encryption, rfc9180, cryptography, http, e2e, chacha20, x25519, fastapi, aiohttp, httpx, sse, streaming

# hpke-http

End-to-end encryption for HTTP APIs using RFC 9180 HPKE (Hybrid Public Key Encryption). Drop-in middleware for FastAPI, aiohttp, and httpx.

[![CI](https://github.com/dualeai/hpke-http/actions/workflows/test.yml/badge.svg)](https://github.com/dualeai/hpke-http/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/hpke-http)](https://pypi.org/project/hpke-http/)
[![Downloads](https://img.shields.io/pypi/dm/hpke-http)](https://pypi.org/project/hpke-http/)
[![Python](https://img.shields.io/pypi/pyversions/hpke-http)](https://pypi.org/project/hpke-http/)
[![License](https://img.shields.io/pypi/l/hpke-http)](https://opensource.org/licenses/Apache-2.0)

## Highlights

- **Transparent** - Drop-in middleware, no application code changes
- **End-to-end encryption** - Protects data even when TLS terminates at CDN or load balancer
- **PSK binding** - Each request cryptographically bound to pre-shared key (API key)
- **Replay protection** - Counter-based nonces prevent replay attacks
- **RFC 9180 compliant** - Auditable, interoperable standard
- **Memory-efficient** - Streams large file uploads with O(chunk_size) memory

## Installation

```bash
uv add "hpke-http[fastapi]"       # Server
uv add "hpke-http[aiohttp]"       # Client (aiohttp)
uv add "hpke-http[httpx]"         # Client (httpx)
uv add "hpke-http[fastapi,zstd]"  # + zstd compression (gzip fallback included)
```

## Quick Start

Standard JSON requests, SSE (Server-Sent Events) streaming, and file uploads are transparently encrypted.

### Server (FastAPI)

```python
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from hpke_http.middleware.fastapi import HPKEMiddleware
from hpke_http.constants import KemId

app = FastAPI()

async def resolve_psk(scope: dict) -> tuple[bytes, bytes]:
    # Get derived PSK ID from X-HPKE-PSK-ID header (already decoded)
    psk_id = scope.get("hpke_psk_id")
    # Look up API key by its derived ID (see "PSK Authentication" section)
    record = await db.lookup_by_derived_id(psk_id)  # Returns {psk, tenant_id}
    scope["tenant_id"] = record["tenant_id"]  # For authorization
    return (record["psk"], psk_id)

app.add_middleware(
    HPKEMiddleware,
    private_keys={KemId.DHKEM_X25519_HKDF_SHA256: private_key},
    psk_resolver=resolve_psk,
)

@app.post("/users")
async def create_user(request: Request):
    data = await request.json()  # Decrypted by middleware
    return {"id": 123, "name": data["name"]}  # Encrypted by middleware

@app.get("/users/{user_id}")
async def get_user(request: Request):
    return {"id": 123, "name": "Alice"}  # Encrypted by middleware

@app.post("/chat")
async def chat(request: Request):
    data = await request.json()

    async def generate():
        yield b"event: progress\ndata: {\"step\": 1}\n\n"
        yield b"event: complete\ndata: {\"result\": \"done\"}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")
```

### Client (aiohttp)

```python
import hashlib
import aiohttp
from hpke_http.middleware.aiohttp import HPKEClientSession

# Derive PSK ID from API key (see "PSK Authentication" section)
psk_id = hashlib.sha256(api_key).digest()

async with HPKEClientSession(
    base_url="https://api.example.com",
    psk=api_key,        # >= 32 bytes
    psk_id=psk_id,      # Derived from key, not tenant ID
    # compress=True,           # Compression (zstd preferred, gzip fallback)
    # require_encryption=True, # Raise if server responds unencrypted
    # release_encrypted=True,  # Free encrypted bytes after decryption (saves memory)
) as session:
    # POST with JSON body
    async with session.post("/users", json={"name": "Alice"}) as resp:
        user = await resp.json()

    # SSE streaming
    async with session.post("/chat", json={"prompt": "Hello"}) as resp:
        async for chunk in session.iter_sse(resp):
            print(chunk)  # b"event: progress\ndata: {...}\n\n"

    # GET (bodyless) - response is still encrypted
    async with session.get("/users/123") as resp:
        user = await resp.json()

    # File upload - streams with O(chunk_size) memory
    form = aiohttp.FormData()
    form.add_field("file", open("large.pdf", "rb"), filename="large.pdf")
    async with session.post("/upload", data=form) as resp:
        result = await resp.json()
```

### Client (httpx)

```python
import hashlib
from hpke_http.middleware.httpx import HPKEAsyncClient

# Derive PSK ID from API key (see "PSK Authentication" section)
psk_id = hashlib.sha256(api_key).digest()

async with HPKEAsyncClient(
    base_url="https://api.example.com",
    psk=api_key,        # >= 32 bytes
    psk_id=psk_id,      # Derived from key, not tenant ID
    # compress=True,           # Compression (zstd preferred, gzip fallback)
    # require_encryption=True, # Raise if server responds unencrypted
    # release_encrypted=True,  # Free encrypted bytes after decryption (saves memory)
) as client:
    # POST with JSON body
    resp = await client.post("/users", json={"name": "Alice"})
    user = resp.json()

    # SSE streaming
    resp = await client.post("/chat", json={"prompt": "Hello"})
    async for chunk in client.iter_sse(resp):
        print(chunk)  # b"event: progress\ndata: {...}\n\n"

    # GET (bodyless) - response is still encrypted
    resp = await client.get("/users/123")
    user = resp.json()

    # File upload - streams with O(chunk_size) memory
    resp = await client.post("/upload", files={"file": open("large.pdf", "rb")})
    result = resp.json()
```

## Documentation

- [RFC 9180 - HPKE](https://datatracker.ietf.org/doc/rfc9180/)
- [RFC 7748 - X25519](https://datatracker.ietf.org/doc/rfc7748/)
- [RFC 5869 - HKDF](https://datatracker.ietf.org/doc/rfc5869/)
- [RFC 8439 - ChaCha20-Poly1305](https://datatracker.ietf.org/doc/rfc8439/)
- [RFC 8878 - Zstandard](https://datatracker.ietf.org/doc/rfc8878/) (preferred compression)
- [RFC 1952 - Gzip](https://datatracker.ietf.org/doc/rfc1952/) (fallback compression, always available)
- [RFC 9110 - HTTP Semantics](https://datatracker.ietf.org/doc/rfc9110/) (Accept-Encoding negotiation)

## Cipher Suite

| Component | Algorithm | ID |
| --------- | --------- | ------ |
| KEM (Key Encapsulation) | DHKEM(X25519, HKDF-SHA256) | 0x0020 |
| KDF (Key Derivation) | HKDF-SHA256 | 0x0001 |
| AEAD (Authenticated Encryption) | ChaCha20-Poly1305 | 0x0003 |
| Mode | PSK (Pre-Shared Key) | 0x01 |

## PSK Authentication

HPKE PSK mode binds each request to a pre-shared key. This requires two values:

| Value | What it is | Example |
|-------|------------|---------|
| **PSK** | The secret key material | API key bytes, `b"sk_live_7f3a9c..."` |
| **PSK ID** | Identifies *which* PSK to use | `SHA256(api_key)` — 32 bytes recommended, min 1 byte |

> **Data model:** One tenant typically has *many* API keys (dev/prod, per-service, per-team-member). The PSK ID identifies the specific key, not the tenant.

### Security Considerations

[RFC 9180 §9.4](https://www.rfc-editor.org/rfc/rfc9180.html#section-9.4) warns that `psk_id` **"might be considered sensitive, since, in a given application context, [it] might identify the sender."**

The `X-HPKE-PSK-ID` header is sent in plaintext (only base64url-encoded, not encrypted). [RFC 9257](https://www.rfc-editor.org/rfc/rfc9257.html) documents the risks:

| Risk | Description |
|------|-------------|
| **Passive linkability** | Observers correlate connections using the same PSK ID |
| **Traffic analysis** | Identify specific API keys/users by their identifier |
| **Active suppression** | Targeted blocking based on observed identifiers |

### Mitigation: Derive PSK ID from the Key

**Derive `psk_id` from the PSK itself** ([RFC 9180 §9.4](https://www.rfc-editor.org/rfc/rfc9180.html#section-9.4)):

```mermaid
sequenceDiagram
    participant C as Client
    participant S as Server

    Note over C: psk_id = SHA256(psk)
    C->>C: Encrypt body with (psk, psk_id)
    C->>S: POST /api<br/>X-HPKE-PSK-ID: <derived_id>
    S->>S: Lookup PSK by derived_id
    S->>S: Decrypt with (psk, psk_id)
    S-->>C: Encrypted response
```

### Implementation

**Client** — derive PSK ID from key:

```python
import hashlib

api_key = b"sk_live_7f3a9c..."  # Your API key (>= 32 bytes)
# Derive PSK ID from the key itself
psk_id = hashlib.sha256(api_key).digest()

async with HPKEClientSession(
    base_url="https://api.example.com",
    psk=api_key,
    psk_id=psk_id,
) as client:
    await client.post("/api", json=data)
```

**Server** — store derived ID when key created, lookup on request:

```python
import hashlib

# Key creation: store derived_id → {psk, tenant_id}
derived_id = hashlib.sha256(api_key).digest()
db.store(derived_id, {"psk": api_key, "tenant_id": tenant_id})

# psk_resolver: lookup by derived_id from header
async def resolve_psk(scope: dict) -> tuple[bytes, bytes]:
    derived_id = scope.get("hpke_psk_id")
    record = await db.lookup(derived_id)
    scope["tenant_id"] = record["tenant_id"]
    return (record["psk"], derived_id)
```

## Wire Format

### Request/Response (Chunked Binary)

See [Header Modifications](#header-modifications) for when headers are added.

```text
Headers:
  X-HPKE-Enc: <base64url(32B ephemeral key)>
  X-HPKE-Stream: <base64url(4B session salt)>
  X-HPKE-PSK-ID: <base64url(derived key ID, 32B recommended)>

Body (repeating chunks):
┌───────────┬────────────┬─────────────────────────────────┐
│ Length(4B)│ Counter(4B)│ Ciphertext (N + 16B tag)        │
│ big-endian│ big-endian │ encrypted: encoding_id || data  │
└───────────┴────────────┴─────────────────────────────────┘
Overhead: 24B/chunk (4B length + 4B counter + 16B tag)
```

### SSE Event

```text
event: enc
data: <base64(counter_be32 || ciphertext)>
Decrypted: raw SSE chunk (e.g., "event: progress\ndata: {...}\n\n")
```

Uses standard base64 (not base64url) - SSE data fields allow +/= characters.

## Compression (Optional)

Zstd reduces bandwidth by **40-95%** for JSON/text. Enable with `compress=True` on both client and server. Payloads < 64 bytes skip compression. See [Compression table](#compression) for algorithm priority.

## Pitfalls

```python
# PSK too short
HPKEClientSession(psk=b"short", psk_id=...)     # InvalidPSKError
HPKEClientSession(psk=secrets.token_bytes(32), psk_id=...)  # >= 32 bytes

# PSK ID must be derived from the key (see "PSK Authentication" section)
psk_id = hashlib.sha256(api_key).digest()
HPKEClientSession(psk=api_key, psk_id=psk_id)   # Correct

# SSE missing content-type (won't use SSE format)
return StreamingResponse(gen())                                  # Binary format (wrong for SSE)
return StreamingResponse(gen(), media_type="text/event-stream")  # SSE format (correct)

# Standard responses work automatically - no special handling needed
return {"data": "value"}  # Auto-encrypted as binary chunks
```

## Limits

| Resource | Limit | Applies to |
| -------- | ----- | ---------- |
| HPKE messages/context | 2^96-1 | All |
| Chunks/session | 2^32-1 | All |
| PSK minimum | 32 bytes | All |
| PSK ID minimum | 1 byte | All |
| Chunk size | 64KB | All |
| Binary chunk overhead | 24B (length + counter + tag) | Requests & standard responses |
| SSE event buffer | 64MB (configurable) | SSE only |

> **Note:** SSE is text-only (UTF-8). Binary data must be base64-encoded (+33% overhead).

## HTTP Compatibility

### Protocol Support

| Feature | Supported | Notes |
| ------- | --------- | ----- |
| HTTP/1.1 | Yes | Chunked transfer encoding for streaming |
| HTTP/2 | Yes | Native framing (chunked encoding forbidden by spec) |
| HTTP/3 | Yes | QUIC streams, same semantics as HTTP/2 |
| WebSockets | No | Different protocol, not applicable |

### HTTP Methods

HPKE key exchange happens on every request, including bodyless methods like GET and DELETE.

| Method | Typical Use | Request Body | Response |
| ------ | ----------- | ------------ | -------- |
| POST | Create | Encrypted | Encrypted |
| PUT | Replace | Encrypted | Encrypted |
| PATCH | Update | Encrypted | Encrypted |
| DELETE | Remove | Encrypted (if body) | Encrypted |
| GET | Read | No body | Encrypted |
| HEAD | Metadata | No body | Headers only (no body per HTTP spec) |
| OPTIONS | Preflight | No body | Encrypted |

### Response Encryption (Server)

| Content-Type | Wire Format | Memory |
| ------------ | ----------- | ------ |
| Any non-SSE | Length-prefixed 64KB chunks | O(64KB) buffer |
| `text/event-stream` | Base64 SSE events | O(event size) |

### Response Decryption (Client)

| Content-Type | API | Memory | Delivery |
| ------------ | --- | ------ | -------- |
| Any non-SSE | `resp.json()`, `resp.content` | O(response size) | After full download |
| `text/event-stream` | `async for chunk in iter_sse(resp)` | O(event size) | As events arrive |

> Use `release_encrypted=True` to free encrypted buffer after decryption (reduces peak memory).

### Compression

| Algorithm | Request | Response | Priority |
| --------- | ------- | -------- | -------- |
| Zstd (RFC 8878) | Yes | Yes | 1 (preferred) |
| Gzip (RFC 1952) | Yes | Yes | 2 (fallback) |
| Identity | Yes | Yes | 3 (no compression) |

Auto-negotiated via `Accept-Encoding` header on discovery endpoint (`/.well-known/hpke-keys`).

#### Why HTTP-Level Compression Doesn't Help

Disable gzip/brotli on CDN/LB for HPKE endpoints. Ciphertext is incompressible—HTTP compression wastes CPU. Use `compress=True` on the client instead (compresses before encryption).

## Encryption Scope

### What IS Encrypted

| Component | Encrypted | Format |
| --------- | --------- | ------ |
| Request body | Yes | Binary chunks |
| Response body | Yes | Binary chunks or SSE events |

### What is NOT Encrypted

| Component | Visible to | Reason |
| --------- | ---------- | ------ |
| URL path | Network | Routing requires plaintext |
| Query parameters | Network | Part of URL |
| HTTP method | Network | Protocol requirement |
| HTTP headers | Network | Routing, caching, auth |
| Status code | Network | Protocol requirement |
| TLS metadata | Network | Transport layer |

### Header Modifications

| Header | Request | Response | Reason |
| ------ | ------- | -------- | ------ |
| `Content-Type` | Set to `application/octet-stream` (if body) | Preserved | Encrypted body is binary |
| `Content-Length` | Auto (chunked, if body) | Removed | Size changes after encryption |
| `X-HPKE-Enc` | Always | - | Ephemeral public key |
| `X-HPKE-Stream` | Always | Added | Session salt for nonces |
| `X-HPKE-PSK-ID` | Always | - | Derived PSK identifier (see [PSK Authentication](#psk-authentication)) |
| `X-HPKE-Encoding` | Added (if compressed) | - | Compression algorithm |
| `X-HPKE-Content-Type` | Added (if body) | - | Original Content-Type for server parsing |

### Security Boundary

```
┌─────────────────────────────────────────────────────────────┐
│ TLS Encrypted (transport)                                   │
│  ┌───────────────────────────────────────────────────────┐  │
│  │ HTTP Layer (visible to CDN/LB/proxies)                │  │
│  │  • Method: POST                                       │  │
│  │  • URL: /api/chat                                     │  │
│  │  • Headers: Authorization, X-HPKE-*, Content-Type     │  │
│  │  ┌─────────────────────────────────────────────────┐  │  │
│  │  │ HPKE Encrypted (end-to-end)                     │  │  │
│  │  │  • Request body: {"prompt": "Hello"}            │  │  │
│  │  │  • Response body: {"response": "Hi!"}           │  │  │
│  │  │  • SSE events: event: done\ndata: {...}\n\n     │  │  │
│  │  └─────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

## Low-Level API

Direct access to HPKE seal/open operations:

```python
from hpke_http.hpke import seal_psk, open_psk

# pk_r: recipient public key, sk_r: recipient secret key
# psk/psk_id: pre-shared key and identifier, aad: additional authenticated data
enc, ct = seal_psk(pk_r, b"info", psk, psk_id, b"aad", b"plaintext")
pt = open_psk(enc, sk_r, b"info", psk, psk_id, b"aad", ct)
```

## Security

Uses OpenSSL constant-time implementations via `cryptography` library.

- [Security Policy](./SECURITY.md) - Vulnerability reporting
- [SBOM](https://github.com/dualeai/hpke-http/releases) - Software Bill of Materials (CycloneDX format) attached to releases

## Contributing

Contributions welcome! Please open an issue first to discuss changes.

```bash
make install      # Setup venv
make test         # Run tests
make lint         # Format and lint
```

## License

[Apache-2.0](https://opensource.org/licenses/Apache-2.0)