encypher-ai 3.0.6
Embed invisible metadata in AI-generated text using zero-width characters.
Requires Python: >=3.9

<p align="center">
  <img src="docs/assets/horizontal-logo.png" alt="Encypher Logo" width="600">
</p>

# Encypher Core

[![PyPI version](https://img.shields.io/pypi/v/encypher-ai.svg)](https://pypi.org/project/encypher-ai/)
[![GitHub stars](https://img.shields.io/github/stars/encypherai/encypher-ai.svg?style=social&label=Star)](https://github.com/encypherai/encypher-ai)
[![Python versions](https://img.shields.io/pypi/pyversions/encypher-ai.svg)](https://pypi.org/project/encypher-ai/)
[![Documentation](https://img.shields.io/badge/docs-docs.encypherai.com-blue)](https://docs.encypherai.com)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![Build Status](https://github.com/encypherai/encypher-ai/actions/workflows/python-package.yml/badge.svg)](https://github.com/encypherai/encypher-ai/actions/workflows/python-package.yml)

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![Linter: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

A Python package for embedding and extracting metadata in text using Unicode variation selectors without affecting readability.

## Overview

Encypher Core provides tools for invisibly encoding metadata (such as model information, timestamps, and custom data) into text generated by AI models. This enables:

- **Provenance tracking**: Identify which AI model generated a piece of text
- **Timestamp verification**: Know when text was generated
- **C2PA-Compatible Manifests**: Embed manifests inspired by the C2PA standard, with support for CBOR encoding for maximum interoperability.
- **Custom metadata**: Embed any additional information you need
- **Tamper detection using digital signatures**: Verify text integrity using digital signatures
- **Streaming support**: Works with both streaming and non-streaming LLM outputs
- **LLM integrations**: Ready-to-use integrations with OpenAI, Google Gemini, Anthropic Claude, and more
- **Modular architecture**: Clean separation of key management, payload handling, and signing operations

The encoding is done using Unicode variation selectors, which are designed to specify alternative forms of characters without affecting text appearance or readability.

## Relationship with C2PA

Encypher implements the [Coalition for Content Provenance and Authenticity (C2PA)](https://c2pa.org/) standard for plain-text environments using the **C2PA Text Embedding Standard**. The core library uses the [`c2pa-text`](https://github.com/encypherai/c2pa-text) reference implementation for text manifest wrappers.

Specification reference:
https://spec.c2pa.org/specifications/specifications/2.3/specs/C2PA_Specification.html#embedding_manifests_into_unstructured_text

This project is maintained by Encypher. The maintainers of this codebase are co-chairs of the C2PA Text Task Force and authors of the linked specification.

Encypher relies on the MIT-licensed `encypherai/c2pa-text` package for C2PA Text Manifest wrapper encoding/decoding (the wrapper transport layer), while `encypher-ai` provides end-to-end signing, embedding, extraction, and verification.

Key features:
- **C2PA 2.3 compliant manifests** with proper structure (`@context`, `instance_id`, `claim_generator`)
- **Hard binding** via `c2pa.hash.data.v1` assertions with NFC normalization
- **Soft binding** via `c2pa.soft_binding.v1` for manifest recovery
- **C2PATextManifestWrapper** encoding using Unicode variation selectors (U+FE00-U+FE0F, U+E0100-U+E01EF)
- **COSE_Sign1 signatures** for cryptographic integrity
- **Format-agnostic embedding** that works across JSON, XML, databases, and message brokers
- **Spec version**: Aligned with C2PA 2.3 specification (unstructured text embedding)

Our implementation uses the official `c2pa-text` library to encode C2PA manifests into invisible Unicode variation selectors, enabling provenance tracking and tamper detection without altering the visible appearance of the text.

Learn more about [Encypher's relationship with C2PA](https://docs.encypherai.com/package/user-guide/c2pa-relationship/) in our documentation.

### C2PA `@context` compatibility configuration (v3.0.3+)

`encypher-ai` allows production services to control the emitted and accepted C2PA schema contexts via environment variables:

- `ENCYPHER_C2PA_CONTEXT_URL` controls the emitted `@context` during embedding.
- `ENCYPHER_C2PA_ACCEPTED_CONTEXTS` controls the verifier allowlist (JSON list or comma-separated string).

### Maintenance & Support

While this library is open source, Encypher offers an **Enterprise API** for:

- Managing cryptographic keys at scale (including HSM-backed workflows)
- Automated verification, revocation, and trust anchor management
- Content production workflows for signing at scale
- Analytics and tracking for signed content

[Learn more about Encypher Enterprise](https://encypherai.com)

### Cryptography and COSE

- Native COSE_Sign1 EdDSA (Ed25519) implementation using `cbor2` + `cryptography`.
- No runtime dependency on external COSE libraries; `pycose` has been removed.
- Security hygiene: removes the transitive `python-ecdsa` dependency (GHSA-wj6h-64fc-37mp) from the dependency graph.

## LLM Integrations

Encypher seamlessly integrates with popular LLM providers:

- **OpenAI**: GPT-3.5, GPT-4o, and other OpenAI models
- **Google Gemini**: Gemini 2.0 Flash, Pro, and other Gemini models
- **Anthropic Claude**: Claude 3 Opus, Sonnet, Haiku, and other Claude models
- **LiteLLM**: For unified access to multiple LLM providers

Check our [documentation](https://docs.encypherai.com/package/integration/) for detailed integration examples and code snippets for each provider.

## Demo Video

[![Encypher Demo Video](https://img.youtube.com/vi/_MNP0nHc77k/0.jpg)](https://www.youtube.com/watch?v=_MNP0nHc77k)

Watch our demo video to see Encypher in action, demonstrating how to embed and verify metadata in AI-generated content.

## Interactive Colab Notebook

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MnGzqf0VmPgw8fOXxUgBKKwgGBET4PYw?usp=sharing)

Try Encypher directly in your browser with our interactive Google Colab notebook. The notebook demonstrates all the core features including metadata embedding, extraction, digital signature verification, and tampering detection.

### Local Jupyter Notebook Example

For a local demonstration, check out the detailed Jupyter Notebook example included in the repository:
[`encypher/examples/encypher_v2_demo.ipynb`](./encypher/examples/encypher_v2_demo.ipynb)

This notebook covers key generation, basic and manifest format usage, and tamper detection using the latest version (v2.2.0+).

## Installation

Use `uv` for installation and dependency management:

```bash
# Add to an existing project
uv add encypher-ai

# Or run ad-hoc scripts
uv run python -c "import encypher; print(encypher.__version__)"
```

## Quick Start

> **Note:** Digital signatures require managing a private/public key pair. You can use the helper script `encypher/examples/generate_keys.py` to create your first key pair and get setup instructions, or generate keys programmatically as shown below.

> **Note:** Timestamp is optional across all formats (basic, manifest, cbor_manifest, c2pa). If omitted, verification still works. For C2PA, action assertions (e.g., `c2pa.created`, `c2pa.watermarked`) omit the `when` field when no timestamp is supplied.

### Basic Encoding and Verification

```python
from encypher.core.unicode_metadata import UnicodeMetadata
from encypher.core.keys import generate_ed25519_key_pair # Updated to specific key type
from cryptography.hazmat.primitives.asymmetric.types import Ed25519PublicKey, Ed25519PrivateKey
from typing import Optional, Dict, Union # Added Union
import time
from encypher.core.payloads import BasicPayload, ManifestPayload # For type hinting verified_payload

# --- Key Management (Replace with your actual key management) ---
# Generate a new Ed25519 key pair
private_key: Ed25519PrivateKey
public_key: Ed25519PublicKey
private_key, public_key = generate_ed25519_key_pair()
signer_id_example = "readme-signer-001" # Using signer_id

# Store public keys (e.g., in a database or secure store)
public_keys_store: Dict[str, Ed25519PublicKey] = { signer_id_example: public_key }

# Create a provider function to look up public keys by ID
def public_key_provider(signer_id: str) -> Optional[Ed25519PublicKey]: # Renamed and uses signer_id
    return public_keys_store.get(signer_id)
# -----------------------------------------------------------------

# Core information for embedding
current_timestamp = int(time.time())  # Current Unix timestamp (seconds since epoch)

# Custom metadata payload (user-defined data)
custom_payload = {
    "model_id": "gpt-4o-2024-05-13",
    "source_script": "README_quickstart",
    "user_defined_version": "2.3.0" # Updated version
}

# Embed metadata and sign
# The 'metadata_format' and 'version' (Encypher spec version) parameters for embed_metadata
# default to "basic" and the latest spec version respectively.
encoded_text = UnicodeMetadata.embed_metadata(
    text="This is a sample text generated by an AI model.",
    private_key=private_key,         # Private key for signing
    signer_id=signer_id_example,     # Identifier for the key pair
    custom_metadata=custom_payload,  # Your arbitrary metadata
    omit_keys=["user_id", "session_id"],  # Example of redacting fields
)

# Extract metadata (without verification - returns the raw payload if successful)
# This is useful for quick inspection but does not guarantee authenticity or integrity.
extracted_unverified_payload = UnicodeMetadata.extract_metadata(encoded_text)
print(f"Extracted (unverified) payload: {extracted_unverified_payload}")

# Verify the signature and extract metadata using the public key provider
# This is the recommended way to get trusted metadata.
is_valid: bool
extracted_signer_id: Optional[str]
verified_payload: Union[BasicPayload, ManifestPayload, None] # Type hint for clarity
is_valid, extracted_signer_id, verified_payload = UnicodeMetadata.verify_metadata(
    text=encoded_text,
    public_key_provider=public_key_provider
)

print(f"\nSignature valid: {is_valid}")
if is_valid and verified_payload:
    print(f"Verified Signer ID: {extracted_signer_id}")
    print(f"Verified Timestamp: {verified_payload.timestamp}")
    print(f"Verified Custom Metadata: {verified_payload.custom_metadata}")
    print(f"Verified Format: {verified_payload.format}")
    print(f"Verified Encypher Spec Version: {verified_payload.version}")
else:
    print("Metadata could not be verified or extracted.")
```

#### Embedding without a timestamp

```python
encoded_text_no_ts = UnicodeMetadata.embed_metadata(
    text="This is a sample text generated by an AI model.",
    private_key=private_key,
    signer_id=signer_id_example,
    custom_metadata=custom_payload,
)

is_valid2, _, payload2 = UnicodeMetadata.verify_metadata(
    text=encoded_text_no_ts,
    public_key_provider=public_key_provider
)
print(f"Valid: {is_valid2}; Timestamp present: {bool(getattr(payload2, 'timestamp', None))}")
```

### Streaming Example

```python
from encypher.streaming.handlers import StreamingHandler
from encypher.core.unicode_metadata import UnicodeMetadata # Added for verification
from encypher.core.keys import generate_ed25519_key_pair # Updated to specific key type
from cryptography.hazmat.primitives.asymmetric.types import Ed25519PublicKey, Ed25519PrivateKey
from typing import Optional, Dict, Union # Added Union
import time
from encypher.core.payloads import BasicPayload, ManifestPayload # For type hinting verified_payload

# --- Assuming key setup from the 'Basic Encoding and Verification' example ---

# Custom metadata for streaming example
stream_timestamp = int(time.time())
stream_custom_payload = {
    "model_id": "gpt-4o-2024-05-13",
    "source_script": "README_streaming_example",
    "user_defined_version": "2.3.0" # Updated version
}

# Create a streaming handler
handler = StreamingHandler(
    private_key=private_key,
    signer_id=signer_id_example,
    timestamp=stream_timestamp,
    custom_metadata=stream_custom_payload,
    # metadata_format defaults to "basic" (also accepts "manifest", "cbor_manifest", or "jumbf")
    # encode_first_chunk_only defaults to True, which is common for streaming
)

chunks = [
    "This is ",
    "a sample ",
    "text generated ",
    "by an AI model, delivered in chunks."
]

full_response_from_stream = ""
print("\nSimulating stream output:")
for chunk in chunks:
    processed_chunk = handler.process_chunk(chunk) # Aligned to process_chunk
    if processed_chunk: # process_chunk might return None if it only buffers
        print(processed_chunk, end="")
        full_response_from_stream += processed_chunk

# Complete the stream (important for final metadata embedding if not all chunks were processed)
final_chunk = handler.finalize()
if final_chunk:
    print(final_chunk, end="")
    full_response_from_stream += final_chunk
print("\n--- End of Stream ---")

# Verify the full streamed text
# For streamed content, hard binding is not added, so we must disable it during verification.
print(f"\nVerifying full streamed text: '{full_response_from_stream[:50]}...' ({len(full_response_from_stream)} chars)")
is_stream_valid: bool
stream_signer_id: Optional[str]
stream_payload: Union[BasicPayload, ManifestPayload, None]
is_stream_valid, stream_signer_id, stream_payload = UnicodeMetadata.verify_metadata(
    text=full_response_from_stream,
    public_key_provider=public_key_provider, # Using the provider from basic example
    require_hard_binding=False # Disable for streaming
)

print(f"\nStream signature valid: {is_stream_valid}")
if is_stream_valid and stream_payload:
    print(f"Stream Verified Signer ID: {stream_signer_id}")
    print(f"Stream Verified Timestamp: {stream_payload.timestamp}")
    print(f"Stream Verified Custom Metadata: {stream_payload.custom_metadata}")
    print(f"Stream Verified Format: {stream_payload.format}")
    print(f"Stream Verified Encypher Spec Version: {stream_payload.version}")
else:
    print("Stream metadata could not be verified or extracted.")
```

## Key Features

- **Invisible Metadata**: Embed metadata in text without affecting its visible appearance or readability.
- **Digital Signatures**: Cryptographically sign metadata to ensure authenticity and detect tampering.
- **Streaming Support**: Process and embed metadata in real-time as text is generated or streamed.
- **Customizable Metadata**: Embed any JSON-serializable information relevant to your application.
- **Modular Architecture**: Clean separation of key management, payload handling, and signing operations.

## Command Line Interface

Encypher includes a command-line interface for quick encoding and decoding tasks.

First, ensure you have generated a key pair. You can use the `generate-keys` command for this:

```bash
# Generate a new Ed25519 key pair, saving public key as my_signer_id.pem
python -m encypher.examples.cli_example generate-keys --output-dir ./keys --signer-id my_signer_id
```
This will create `private_key.pem` and `keys/my_signer_id.pem` (if `--output-dir ./keys` was used and `my_signer_id` was the signer ID).

### Encoding Text with Metadata

To encode text with metadata:

```bash
# Basic encoding
python -m encypher.examples.cli_example encode \
  --text "This is a sample text generated by an AI model." \
  --private-key ./keys/private_key.pem \
  --signer-id my_signer_id
```

Additional options:
*   `--output-file`: Optional. File to save encoded text; otherwise, prints to stdout.
*   `--custom-metadata`: Optional. A JSON string for your custom data (e.g., `'{\"key\": \"value\"}'`).
*   `--timestamp`: Optional. Integer Unix timestamp. Defaults to current time.
*   `--model-id`: Optional. Convenience for adding a model ID to custom metadata.
*   `--omit-keys`: Optional. Space separated list of metadata keys to omit before signing.

### Decoding and Verifying Metadata

To extract and verify metadata from text:

```bash
# Basic decoding
python -m encypher.examples.cli_example decode \
  --text "Text with embedded metadata..." \
  --public-key-dir ./keys
```

Additional options:
*   `--output-file`: Optional. File to save decoded metadata; otherwise, prints to stdout.
*   `--verify`: Optional. Verify the signature (default: True).

## Development Setup

To set up the project for development:

```bash
# Install uv (if not already installed)
pip install uv

# Install Encypher in editable mode for development
uv pip install -e .
```

## Migration Guide

If you're upgrading from a previous version, check out our [Migration Guide](https://docs.encypherai.com/package/user-guide/migration-guide/) to learn about the new modular structure introduced in version 2.2.0.

## Advanced Usage: C2PA-Inspired Manifests

Encypher supports embedding structured manifests inspired by the C2PA standard, providing robust provenance and tamper detection for text content. This is ideal for tracking the origin and history of AI-generated text.

The library handles the creation of the manifest, including calculating the content hash of the original, un-encoded text, and bundles it into a CBOR-encoded, signed payload.

> **Note:** The `timestamp` parameter is optional. If you omit it, C2PA actions created by Encypher (e.g., the `c2pa.watermarked` action) will not include a `when` field; behavior is unchanged when a valid timestamp is provided.

### Example: Embedding and Verifying a Manifest

This example demonstrates the end-to-end process:

```python
# --- Imports and Key Setup (from Quick Start) ---
from encypher.core.unicode_metadata import UnicodeMetadata
from encypher.core.keys import generate_ed25519_key_pair
from cryptography.hazmat.primitives.asymmetric.types import Ed25519PublicKey, Ed25519PrivateKey
from typing import Optional, Dict, Union
import time
from encypher.core.payloads import BasicPayload, ManifestPayload

# Generate a new Ed25519 key pair
private_key: Ed25519PrivateKey
public_key: Ed25519PublicKey
private_key, public_key = generate_ed25519_key_pair()
signer_id_manifest = "manifest-signer-001"

# Store public keys and create a provider function
public_keys_store: Dict[str, Ed25519PublicKey] = { signer_id_manifest: public_key }
def public_key_provider(signer_id: str) -> Optional[Ed25519PublicKey]:
    return public_keys_store.get(signer_id)
# ----------------------------------------------------

# Original text to be signed
original_text = "This is an important statement generated by an AI assistant."

# Embed a C2PA-inspired manifest
# The library automatically calculates the content hash of the original text.
encoded_text_manifest = UnicodeMetadata.embed_metadata(
    text=original_text,
    private_key=private_key,
    signer_id=signer_id_manifest,
    timestamp=int(time.time()),
    metadata_format="cbor_manifest",  # Use the CBOR manifest format ("jumbf" is also supported)
    claim_generator="Encypher README Example v2.3",
    actions=[
        {
            "action": "c2pa.created",
            "when": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "description": "Text was created by an AI model."
        }
    ],
    ai_info={
        "model_id": "gpt-4o-2024-05-13",
        "prompt": "Write a short, important statement."
    }
)

print(f"Text with embedded manifest: '{encoded_text_manifest[:60]}...'")

### Verifying the Manifest and Detecting Tampering

Verification confirms both the signature's authenticity and the text's integrity. Any change to the original text will cause verification to fail.

# 1. Verify the original, unmodified text
is_valid, signer, payload = UnicodeMetadata.verify_metadata(
    text=encoded_text_manifest,
    public_key_provider=public_key_provider
)

print(f"\nVerification of original text successful: {is_valid}")
if is_valid and payload:
    print(f"  - Signer ID: {signer}")
    # The payload is a ManifestPayload object, so we can access its attributes
    if isinstance(payload, ManifestPayload):
        print(f"  - Claim Generator: {payload.claim_generator}")
        print(f"  - Actions: {payload.actions}")
    # The content hash is stored inside the manifest and checked automatically
    # during verification.

# 2. Attempt to verify tampered text
tampered_text = encoded_text_manifest.replace("important", "unimportant")

is_tampered_valid, _, _ = UnicodeMetadata.verify_metadata(
    text=tampered_text,
    public_key_provider=public_key_provider
)

print(f"\nVerification of tampered text successful: {is_tampered_valid}") # Expected: False
if not is_tampered_valid:
    print("  - As expected, verification failed, indicating the text was tampered with.")
```

The content hash in our implementation covers only the plain text content:

- Extracts all paragraph text from the document (ignoring HTML markup)
- Computes a SHA-256 hash on the UTF-8 encoded text
- Calculates the hash before any metadata embedding occurs
- Does not include the embedded metadata itself (Unicode variation selectors)

#### How Our Embedding Works

Our embedding engine uses a robust, multi-strategy approach to invisibly attach metadata to text using Unicode variation selectors (e.g., U+FE00-FE0F):

- **Distributed Embedding (Primary)**: To maximize resilience, metadata characters are intelligently interleaved throughout the source text, attached to multiple characters.
- **Append-to-End (Fallback)**: If the text is too short or lacks suitable characters for the primary strategy, the metadata is appended to the end of the string.
- **Content Integrity**: In both cases, the original visible text is preserved, and a "hard-binding" content hash in the manifest ensures its integrity.

This hybrid approach ensures that metadata can be reliably embedded in a wide variety of text content.

For more detailed information, see our [Content Hash and Embedding Technical Guide](./docs/package/technical/content-hash-and-embedding.md) and [Tamper Detection Guide](./docs/package/user-guide/tamper-detection.md).

## License

This project is licensed under the GNU Affero General Public License v3.0 - see the LICENSE file for details. Commercial licensing options are also available - see our [Licensing Guide](https://docs.encypherai.com/package/user-guide/licensing/) for details.