foil-sdk 0.5.0
Foil SDK for monitoring and logging AI model invocations
Requires Python: >=3.8
Keywords: ai, monitoring, logging, openai, llm, opentelemetry, tracing

# Foil Python SDK

Python SDK for monitoring and logging AI agent invocations with Foil. Features distributed tracing, semantic search, custom evaluations, multimodal content support (images, documents), signals/feedback, and OpenAI integration.

## Installation

```bash
pip install foil-sdk
```

Or install from source:

```bash
pip install -e /path/to/foil-sdk
```

## Quick Start

```python
from foil import Foil, create_foil_tracer, SpanKind

# Create a tracer for your agent
tracer = create_foil_tracer(
    api_key="your-api-key",
    agent_name="my-agent",
    base_url="https://api.foil.dev/api",  # or your self-hosted URL
)

# Trace an agent execution
async def my_agent():
    async with tracer.trace("weather-query", input="User asked about weather") as ctx:
        # Create an LLM span
        span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="What is the weather?")

        # ... call your LLM ...

        await span.end(
            output="The weather is sunny today!",
            tokens={"prompt": 10, "completion": 8, "total": 18}
        )

        return "Agent completed"
```

## Table of Contents

- [Distributed Tracing](#distributed-tracing)
- [Semantic Search](#semantic-search)
- [Custom Evaluations](#custom-evaluations)
- [Multimodal Content](#multimodal-content)
- [Signals & Feedback](#signals--feedback)
- [OpenAI Integration](#openai-integration)
- [Experiments (A/B Testing)](#experiments-ab-testing)
- [Low-Level API](#low-level-api)
- [API Reference](#api-reference)
- [Examples](#examples)

## Distributed Tracing

### Using the Tracer

```python
from foil import create_foil_tracer, SpanKind

tracer = create_foil_tracer(
    api_key="your-api-key",
    agent_name="customer-support-agent",
)

async with tracer.trace("support-query", input=user_message, session_id=conversation_id) as ctx:
    # LLM span for the main model call
    span = await ctx.start_span(SpanKind.LLM, "gpt-4-turbo", input=messages)

    response = await openai.chat.completions.create(
        model="gpt-4-turbo",
        messages=messages,
    )

    await span.end(
        output=response.choices[0].message.content,
        tokens={
            "prompt": response.usage.prompt_tokens,
            "completion": response.usage.completion_tokens,
            "total": response.usage.total_tokens,
        }
    )
```

### Span Kinds

```python
from foil import SpanKind

SpanKind.AGENT      # Root agent span
SpanKind.LLM        # Language model calls
SpanKind.TOOL       # Tool/function executions
SpanKind.CHAIN      # Chain of operations
SpanKind.RETRIEVER  # RAG retrieval operations
SpanKind.EMBEDDING  # Embedding model calls
SpanKind.CUSTOM     # Custom operation types
```

### Nested Spans

```python
async with tracer.trace("order-lookup") as ctx:
    # LLM decides to use a tool
    llm_span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="Find order #12345")

    # Tool execution (child of LLM span)
    tool_span = await ctx.start_span(SpanKind.TOOL, "lookup_order", input={"order_id": "12345"})
    result = await database.find_order("12345")
    await tool_span.end(output=result)

    await llm_span.end(output=f"Order found: {result}")
```

## Semantic Search

Search through your traces using natural language queries. Foil automatically embeds your trace content and enables AI-powered semantic search.

### Basic Search

```python
from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Search for conversations about a topic
results = foil.semantic_search("conversations about refund requests")

print(f"Found {len(results['results'])} matching traces")
for result in results["results"]:
    print(f"Trace: {result['traceId']}, Similarity: {result['similarity'] * 100:.1f}%")
```

### Search with Filters

```python
results = foil.semantic_search(
    "user asking about pricing",
    agent_id="customer-support-agent",  # Filter by specific agent
    agent_name="support-bot",           # Or filter by agent name
    from_date="2024-01-01",             # Start date (ISO format)
    to_date="2024-12-31",               # End date (ISO format)
    limit=10,                           # Max results (default: 20)
    offset=0,                           # Pagination offset
    threshold=0.4,                      # Min similarity score 0-1 (default: 0.3)
)
```

### Find Similar Traces

Find traces that are semantically similar to a specific trace:

```python
# Find traces similar to a known trace
similar = foil.find_similar_traces(
    "trace-abc-123",
    limit=5,
    threshold=0.4,
)

print(f"Found {len(similar['results'])} similar traces")
for result in similar["results"]:
    print(f"{result['traceId']}: {result['similarity'] * 100:.1f}% similar")
```

### Check Semantic Search Status

```python
# Get overall embedding status
status = foil.get_semantic_search_status()
print(f"Embedded: {status['embeddedSpans']} / {status['totalSpans']} spans")
print(f"Coverage: {status['coveragePercent']}%")
print(f"Ready: {status['ready']}")

# Get status for specific agent
agent_status = foil.get_semantic_search_status("my-agent-id")
```

## Custom Evaluations

Define custom evaluation criteria to analyze your agent's responses. Evaluations run automatically on new traces and can be boolean, score-based, or categorical.

### Get Evaluation Templates

```python
# List available pre-built evaluation templates
templates = foil.get_evaluation_templates()

for t in templates.get("templates", []):
    print(f"{t['name']}: {t.get('description', 'N/A')}")
```

### Create a Boolean Evaluation

```python
evaluation = foil.create_evaluation("agent-123", {
    "name": "professional_tone",
    "description": "Checks if the response maintains a professional tone",
    "prompt": """Evaluate the assistant's response for professional tone.

Consider:
- Is the language respectful and courteous?
- Does it avoid slang or casual expressions?
- Is it helpful without being condescending?

Return true if professional, false otherwise.""",
    "evaluationType": "boolean",
    "enabled": True,
})

print(f"Created evaluation: {evaluation['evaluation']['name']}")
```

### Create a Score Evaluation (1-10)

```python
evaluation = foil.create_evaluation("agent-123", {
    "name": "helpfulness_score",
    "description": "Rates response helpfulness on a scale of 1-10",
    "prompt": """Rate the helpfulness of the response on a scale of 1-10.

Scoring:
- 1-3: Unhelpful
- 4-5: Somewhat helpful
- 6-7: Helpful
- 8-9: Very helpful
- 10: Exceptional

Return a single number from 1 to 10.""",
    "evaluationType": "score",
    "scoreMin": 1,
    "scoreMax": 10,
    "enabled": True,
})
```

### Create a Category Evaluation

```python
evaluation = foil.create_evaluation("agent-123", {
    "name": "response_intent",
    "description": "Categorizes the type of response",
    "prompt": """Classify the response into one category:

- informational: Provides facts or explanations
- action: Guides through a process
- clarification: Asks for more information
- escalation: Suggests human assistance
- closure: Wraps up the conversation

Return only the category name.""",
    "evaluationType": "category",
    "categories": ["informational", "action", "clarification", "escalation", "closure"],
    "enabled": True,
})
```

### Test an Evaluation

Validate your evaluation prompt before enabling:

```python
result = foil.test_evaluation("agent-123", evaluation_id, {
    "input": "How do I reset my password?",
    "output": "I'd be happy to help! Go to Settings > Security > Reset Password.",
})

print(f"Result: {result['result']}")        # true/false, score, or category
print(f"Reasoning: {result['reasoning']}")  # Explanation
```

### Add Few-Shot Examples

Improve evaluation accuracy with examples:

```python
# Add a positive example
foil.add_evaluation_example("agent-123", evaluation_id, {
    "input": "Why is my order late?",
    "output": "I apologize for the delay. Let me check the status for you.",
    "expectedResult": True,
    "reasoning": "Professional and helpful response",
})

# Add a negative example
foil.add_evaluation_example("agent-123", evaluation_id, {
    "input": "Why is my order late?",
    "output": "idk check tracking yourself",
    "expectedResult": False,
    "reasoning": "Casual and unhelpful",
})
```

### Manage Evaluations

```python
# List all evaluations for an agent
evaluations = foil.get_agent_evaluations("agent-123")

# Get specific evaluation details
evaluation = foil.get_evaluation("agent-123", evaluation_id)

# Update an evaluation
foil.update_evaluation("agent-123", evaluation_id, {
    "description": "Updated description",
    "enabled": False,  # Disable temporarily
})

# Get evaluation analytics
analytics = foil.get_evaluation_analytics("agent-123", evaluation_id)
print(f"Total evaluations: {analytics.get('totalEvaluations', 0)}")
print(f"Distribution: {analytics.get('distribution')}")

# Clone a template
cloned = foil.clone_evaluation_template("agent-123", template_id)

# Remove an example
foil.remove_evaluation_example("agent-123", evaluation_id, example_id)

# Delete an evaluation
foil.delete_evaluation("agent-123", evaluation_id)
```

## Multimodal Content

Foil supports multimodal input/output including images, documents, and other media types.

### Media Categories

```python
from foil import MediaCategory

MediaCategory.IMAGE       # Images (png, jpg, gif, webp, etc.)
MediaCategory.DOCUMENT    # Documents (pdf, doc, docx, etc.)
MediaCategory.SPREADSHEET # Spreadsheets (xlsx, csv, etc.)
MediaCategory.CODE        # Code files
MediaCategory.AUDIO       # Audio files
MediaCategory.VIDEO       # Video files
MediaCategory.ARCHIVE     # Archives (zip, tar, etc.)
MediaCategory.NOTEBOOK    # Jupyter notebooks
MediaCategory.OTHER       # Other file types
```

### Uploading Media

```python
from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Upload from file path
result = foil.upload_media("/path/to/image.png")
print(result["mediaId"])   # 'media_abc123'
print(result["category"])  # 'image'
print(result["filename"])  # 'image.png'
print(result["mimeType"])  # 'image/png'

# Upload from bytes
with open("/path/to/document.pdf", "rb") as f:
    content = f.read()
result = foil.upload_media(
    content,
    filename="document.pdf",
    mime_type="application/pdf",
)

# Upload with trace/span association
result = foil.upload_media(
    "/path/to/image.png",
    trace_id="trace_123",
    span_id="span_456",
    direction="input",  # or "output"
)
```

### Content Blocks

Use content blocks to create multimodal input/output:

```python
from foil import content, ContentBlock, MediaCategory

# Create multimodal content array
multimodal_input = content(
    "Please analyze this image:",
    ContentBlock.media(
        upload_result["mediaId"],
        category=MediaCategory.IMAGE,
        filename="photo.jpg",
        mime_type="image/jpeg",
    ),
    "Focus on the composition and colors."
)
```

### Media Retrieval

```python
# Get media information
media_info = foil.get_media(media_id)

# Get presigned URL for download
url_info = foil.get_media_url(media_id, "original")
print(url_info["url"])  # Presigned S3 URL

# Get extracted content URL (for documents)
extracted_url = foil.get_media_url(media_id, "extracted")

# Batch get multiple media
batch_info = foil.batch_media_info([media_id1, media_id2, media_id3])
```

## Signals & Feedback

Record user feedback and custom signals for your traces:

```python
from foil import Foil

foil = Foil(api_key="your-api-key")

# Record a signal for a specific trace
foil.record_signal({
    "traceId": "trace_123",
    "signalName": "user_satisfaction",
    "value": 4,
    "signalType": "feedback",
    "source": "user",
})

# Batch record signals
foil.record_signal_batch([
    {"traceId": "trace_123", "signalName": "thumbs", "value": True, "source": "user"},
    {"traceId": "trace_123", "signalName": "rating", "value": 5, "source": "user"},
])

# Get signals for a trace
signals = foil.get_trace_signals("trace_123")
```

## OpenAI Integration

Automatic logging for OpenAI calls:

```python
from openai import OpenAI
from foil import Foil

foil = Foil(api_key="your-foil-api-key")
client = foil.wrap_openai(OpenAI())

# All chat completion calls are now automatically logged
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

Supports both streaming and non-streaming responses.

## Experiments (A/B Testing)

Get variant assignments for experiments:

```python
from foil import Foil

foil = Foil(api_key="your-api-key")

# Get assigned variant for a user
assignment = foil.get_experiment_variant("experiment_123", user_id)

if assignment["inExperiment"]:
    print(assignment["variantName"])  # 'control' or 'treatment'
    print(assignment["config"])       # {'model': 'gpt-4', 'temperature': 0.7}
```

## Low-Level API

For more control, use the Foil client directly:

```python
from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Manual span management
trace_id = foil.create_trace_id()
span_id = foil.create_span_id()

foil.start_span({
    "spanId": span_id,
    "traceId": trace_id,
    "name": "gpt-4",
    "agentName": "my-agent",
    "spanKind": "llm",
    "input": messages,
})

# ... do work ...

foil.end_span({
    "spanId": span_id,
    "traceId": trace_id,
    "agentName": "my-agent",
    "output": response,
    "tokens": {"prompt": 100, "completion": 50, "total": 150},
})

# Get trace data
trace = foil.get_trace(trace_id)

# List traces
traces = foil.list_traces(
    agent_name="my-agent",
    limit=10,
    from_date="2024-01-01T00:00:00Z",
)
```

## API Reference

### Foil Client

#### Initialization

```python
foil = Foil(api_key="your-api-key", base_url="https://api.getfoil.ai/api")
```

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `api_key` | str | Yes | Your Foil API key |
| `base_url` | str | No | API base URL (default: https://api.getfoil.ai) |

### Semantic Search Methods

| Method | Description |
|--------|-------------|
| `semantic_search(query, **options)` | Search spans using natural language |
| `find_similar_traces(trace_id, **options)` | Find traces similar to a given trace |
| `get_semantic_search_status(agent_id=None)` | Get embedding statistics |

### Custom Evaluation Methods

| Method | Description |
|--------|-------------|
| `get_evaluation_templates()` | List available evaluation templates |
| `get_agent_evaluations(agent_id)` | List evaluations for an agent |
| `get_evaluation(agent_id, evaluation_id)` | Get evaluation details |
| `create_evaluation(agent_id, data)` | Create a custom evaluation |
| `update_evaluation(agent_id, evaluation_id, data)` | Update an evaluation |
| `delete_evaluation(agent_id, evaluation_id)` | Delete an evaluation |
| `test_evaluation(agent_id, evaluation_id, data)` | Test with sample data |
| `clone_evaluation_template(agent_id, template_id)` | Clone a template |
| `add_evaluation_example(agent_id, evaluation_id, data)` | Add few-shot example |
| `remove_evaluation_example(agent_id, evaluation_id, example_id)` | Remove example |
| `get_evaluation_analytics(agent_id, evaluation_id)` | Get evaluation metrics |

### Media Methods

| Method | Description |
|--------|-------------|
| `upload_media(file, **options)` | Upload media for multimodal content |
| `get_media(media_id, content=None)` | Get media information |
| `get_media_url(media_id, content="original")` | Get presigned download URL |
| `batch_media_info(media_ids)` | Get info for multiple media |

### Signal Methods

| Method | Description |
|--------|-------------|
| `record_signal(data)` | Record a signal |
| `record_signal_batch(signals)` | Record multiple signals |
| `get_trace_signals(trace_id)` | Get signals for a trace |

### Trace Methods

| Method | Description |
|--------|-------------|
| `start_span(data)` | Start a span |
| `end_span(data)` | End a span |
| `get_trace(trace_id)` | Get trace with all spans |
| `list_traces(**options)` | List traces with filters |

### Experiment Methods

| Method | Description |
|--------|-------------|
| `get_experiment_variant(experiment_id, identifier)` | Get variant assignment |

### Legacy Methods (Deprecated)

| Method | Replacement |
|--------|-------------|
| `start_invocation(data)` | Use `start_span(data)` |
| `end_invocation(data)` | Use `end_span(data)` |

## Examples

See the `examples/` directory for complete working examples:

- `semantic_search_example.py` - Semantic search usage
- `custom_evaluations_example.py` - Custom evaluations

Run examples:

```bash
export FOIL_API_KEY=your-api-key
export FOIL_BASE_URL=https://api.getfoil.ai/api
export FOIL_AGENT_ID=your-agent-id

python examples/semantic_search_example.py
python examples/custom_evaluations_example.py
```

## License

MIT