breadcrumb 0.2.14
An automatic, non-intrusive execution tracer for Python for iterative IDE-less debugging and code-flow analysis
License: MIT
Requires Python: >=3.10
Keywords: ai, claude, debugging, execution-tracer, instrumentation, llm, mcp, observability, profiling, tracing

# 🍞 Breadcrumb: Automatic Execution Tracing

**Breadcrumb lets you do this:**

```bash
# Run your code with automatic tracing (NO code changes needed!)
breadcrumb run -c myproject --timeout 60 python my_code.py
```

**Output:**
```bash
============================================================
BREADCRUMB RUN REPORT
============================================================

Key Metrics:
  Total Events: 22
  - Calls: 9
  - Returns: 9
  - Exceptions: 0
  Duration: 4.06 ms
  Status: completed

Call Tree:
-> __main__.<module>
  -> __main__.main
    -> __main__.add
    <- __main__.add (0.05ms)
    -> __main__.multiply
    <- __main__.multiply (0.05ms)
    -> __main__.greet
    <- __main__.greet (0.04ms)
    -> __main__.calculate_total
      -> __main__.multiply
      <- __main__.multiply (0.08ms)
      -> __main__.multiply
      <- __main__.multiply (0.08ms)
      -> __main__.add
      <- __main__.add (0.04ms)
    <- __main__.calculate_total (0.51ms)
  <- __main__.main (1.46ms)
<- __main__.<module> (1.61ms)

Top 10 Most Called Functions:
    __main__.multiply: 3 calls
    __main__.add: 2 calls
    __main__.greet: 1 calls
    __main__.<module>: 1 calls
    __main__.main: 1 calls
    __main__.calculate_total: 1 calls

Recommendations on what to do next:
  Detailed Flow: breadcrumb query -c example01 --flow
  Untraced calls: breadcrumb query -c example01 --gaps
  View specific call: breadcrumb query -c example01 --call <call_id>
     Example 'multiply': breadcrumb query -c example01 --call multiply

```

**Dig in:**
```bash
# Query critical debug information: Called by, calls made, performance stats, input args, return values
breadcrumb query -c example01 --call multiply
```

**Output:**
```json
{
  "function": "__main__.multiply",
  "calls": [
    {
      "function": "__main__.multiply",
      "timestamp": "2025-10-11T22:50:39.444263",
      "args": {
        "a": 7,
        "b": 6
      },
      "return_value": 42,
      "duration_ms": 0.049,
      "called_by": "__main__.main",
      "calls_made": []
    },
    {
      "function": "__main__.multiply",
      "timestamp": "2025-10-11T22:50:39.444732",
      "args": {
        "a": 100,
        "b": 2
      },
      "return_value": 200,
      "duration_ms": 0.082,
      "called_by": "__main__.calculate_total",
      "calls_made": []
    },
    {
      "function": "__main__.multiply",
      "timestamp": "2025-10-11T22:50:39.444900",
      "args": {
        "a": 200,
        "b": 0.15
      },
      "return_value": 30.0,
      "duration_ms": 0.083,
      "called_by": "__main__.calculate_total",
      "calls_made": []
    }
  ]
}
```

```bash
# Find out what was called by traced calls that isn't yet traced by breadcrumb
breadcrumb query -c example01 --gaps
```

**Output:**
```json
{
  "untraced_calls": [
    {
      "function": "json.dumps",
      "module": "json",
      "called_by": "__main__.format_response",
      "call_count": 3,
      "suggested_include": "json.*"
    },
    {
      "function": "requests.post",
      "module": "requests",
      "called_by": "__main__.send_webhook",
      "call_count": 1,
      "suggested_include": "requests.*"
    }
  ],
  "tip": "Add these to your config to trace them:\n  breadcrumb config edit myapp --add-include 'json.*'\n  breadcrumb config edit myapp --add-include 'requests.*'"
}
```

**Expand tracing and re-run:**
```bash
# include untraced calls
breadcrumb config edit --add-include 'requests.*'
# run again
breadcrumb run python my_code.py
breadcrumb query --call post
```

**Output:**
```json
{
  "function": "requests.post",
  "calls": [
    {
      "timestamp": "2025-10-11T15:30:42.123",
      "args": {"url": "https://api.example.com/webhook", "data": {"event": "user.signup"}},
      "return_value": {"status": 200, "body": "OK"},
      "duration_ms": 234.5,
      "called_by": "__main__.send_webhook",
      "calls_made": ["json.dumps"]
    }
  ]
}
```
---

## Why Would You Need This?

**You probably won't. But your AI coding agent will LOVE this.**

### The Problem: AI Agents Can't "Step Through" Code

When humans debug, we:
- Set breakpoints and step through execution
- Watch variables change in real-time
- See the actual call stack in our debugger
- Understand what the code *actually did* vs what we *thought* it would do

When AI agents debug, they:
- ❌ Can't set breakpoints (no interactive debugger access)
- ❌ Can't "watch" execution (no real-time visibility)
- ❌ Can't inspect the call stack (no execution context)
- ❌ Must **guess** what happened by reading static code

**Result:** AI agents spend 80% of debugging time making guesses, reading logs, and asking "what actually ran?"

### The Solution: Breadcrumb = X-Ray Vision for AI Agents

Breadcrumb captures **what actually happened** during execution:
- ✅ **Every function call** with arguments and return values
- ✅ **Exact execution flow** in chronological order
- ✅ **Gaps in coverage** showing untraced function calls
- ✅ **Call relationships** (who called what)
- ✅ **Performance data** (execution duration)

**No code changes. No manual logging. No guessing.**

Just run your code with `breadcrumb run` and query the execution trace with structured commands.

---

## Codex CLI using breadcrumb

```bash
# Prompt
By running 'breadcrumb', present a table showing how long each mathematical function call in the example took.
```

<p align="center">
  <img alt="codex" src="https://raw.githubusercontent.com/AndreRatzenberger/breadcrumb-tracer/main/docs/assets/img/codex_small.png?token=GHSAT0AAAAAADM23FLU3N4KFYFYZTLGL3HQ2HK4HGQ" width="1000">
</p>


## Codex CLI using advanced queries for complex tasks

<p align="center">
  <img alt="codex" src="https://raw.githubusercontent.com/AndreRatzenberger/breadcrumb-tracer/main/docs/assets/img/complex_2.png?token=GHSAT0AAAAAADM23FLVPK2W74UWYKQJTRAK2HK4HIA" width="1000">
</p>




## Key Features

### 🎯 Gap Detection

**Problem:** You can't trace EVERYTHING (performance overhead). So what aren't you tracing?

**Solution:** `breadcrumb query --gaps` shows you exactly which functions were called but not traced.

```bash
breadcrumb query -c myapp --gaps
# Shows: requests.post called 5 times by myapp.api_client
# Suggests: breadcrumb config edit myapp --add-include 'requests.*'
```

**Workflow:** Start minimal → discover gaps → expand tracing → repeat

This is **iterative debugging** - trace only what you need, when you need it.

### 🤖 AI-FIRST Design

**Traditional tools:** Optimize for human convenience (implicit context, smart defaults)
**Breadcrumb:** Optimizes for AI reliability (explicit context, zero ambiguity)

```bash
# ❌ IMPLICIT (confusing for AI agents)
breadcrumb run python script.py          # Which database? Which config?
breadcrumb query --gaps                   # Where are the traces?

# ✅ EXPLICIT (perfect for AI agents)
breadcrumb run -c myapp python script.py  # Clear: using myapp config
breadcrumb query -c myapp --gaps          # Clear: querying myapp database
```

**Result:** Zero context confusion across sessions. AI agents can resume debugging workflows perfectly.

### 📊 Smart Queries (No SQL Required)

AI agents don't need to learn SQL. They just need structured data.

```bash
# Find untraced calls
breadcrumb query -c myapp --gaps

# Show function I/O
breadcrumb query -c myapp --call send_webhook

# Show execution timeline
breadcrumb query -c myapp --flow

# Filter by module
breadcrumb query -c myapp --flow --module myapp

# Search captured events for a literal string (function names, args, payloads, etc.)
breadcrumb query -c myapp --fuzzy breadcrumbs
```

All output is structured JSON, ready for AI consumption.

Call queries accept substrings, so `breadcrumb query -c myapp --call Pizza` resolves to
`__main__.Pizza` (and its `__init__`) and `breadcrumb query -c myapp --call "__main__."`
returns every function in `__main__`. Constructors capture their keyword arguments and
return values automatically, making object creation auditable without extra logging.

Need a human-readable table? Switch formats:

```bash
# Table output with full values preserved for wide SQL results
breadcrumb --format table query -c myapp --disable-truncation \
  "SELECT function_name, data FROM trace_events ORDER BY timestamp DESC LIMIT 5"

# Increase captured payload size when recording new traces (default 2000 chars)
breadcrumb run -c myapp -t 60 --max-chars 5000 python main.py
```

### 🚫 Zero Code Changes

```bash
# ✅ Just run with breadcrumb
breadcrumb run -c myapp --timeout 60 python main.py
# NO code modifications needed!
```

Your codebase stays clean. Tracing is external.

### 🔒 Automatic Secret Redaction

Breadcrumb automatically detects and redacts 50+ sensitive patterns:
- Passwords, API keys, tokens
- Credit cards, SSNs
- JWTs, AWS keys, GitHub tokens

```python
data = {"api_key": "sk-1234567890", "password": "secret"}
# Stored as: {"api_key": "[REDACTED]", "password": "[REDACTED]"}
```

**Safe for production traces.**

### ⚡ Low Overhead

- **PEP 669 backend** (Python 3.12+): < 2% overhead
- **sys.settrace backend** (Python 3.8+): < 5% overhead
- **Selective tracing**: Only trace what you need
- **Async writes**: Non-blocking trace persistence

---

## Quick Start

### Installation

```bash
# Using uv (recommended)
uv pip install -e .

# Or using pip
pip install -e .
```

### The Complete Workflow

**1️⃣ Initialize your project (one-time setup):**
```bash
breadcrumb init myapp
```

Creates:
- `~/.breadcrumb/myapp.yaml` (config with sensible defaults)
- `~/.breadcrumb/myapp-traces.duckdb` (trace database)

**2️⃣ Run your code with tracing:**
```bash
breadcrumb run -c myapp --timeout 60 python main.py
```

Automatically traces execution. NO code changes needed.

**3️⃣ Find gaps in coverage:**
```bash
breadcrumb query -c myapp --gaps
```

Shows which functions were called but not traced.

**4️⃣ Expand tracing based on gaps:**
```bash
breadcrumb config edit myapp --add-include 'requests.*'
breadcrumb config edit myapp --add-include 'sqlalchemy.*'
```

**5️⃣ Re-run with expanded tracing:**
```bash
breadcrumb run -c myapp --timeout 60 python main.py
```

Now traces additional modules!

**6️⃣ Query function details:**
```bash
breadcrumb query -c myapp --call my_function
breadcrumb query -c myapp --call "__main__."    # Substring match, includes constructors
```

Shows args, returns, duration, caller/callee relationships.

**7️⃣ View execution timeline:**
```bash
breadcrumb query -c myapp --flow
```

Shows chronological execution order with call stack depth.

**8️⃣ (Optional) Capture richer payloads:**
```bash
breadcrumb run -c myapp --timeout 60 --max-chars 5000 python main.py
```

Increases argument/return capture size without touching your source code.

---

## AI-FIRST Design Philosophy

### Why `-c PROJECT` is REQUIRED

**All commands require explicit project specification:**

```bash
breadcrumb run -c myapp python script.py       # REQUIRED
breadcrumb query -c myapp --gaps               # REQUIRED
breadcrumb config edit myapp --add-include 'x' # REQUIRED
```

**This eliminates:**
- ❌ "Which database am I querying?"
- ❌ "Which config is active?"
- ❌ "Where did my traces go?"
- ❌ Context loss between AI agent sessions

**Why it matters:**
- **Humans:** "Ugh, typing `-c myapp` every time is annoying!"
- **AI Agents:** "Perfect! No ambiguity, no context confusion!"

**Design philosophy:** Explicit > Implicit for AI workflows.

AI agents don't care about typing extra characters. They care about reliability across sessions.

---

## Command Reference

### Core Commands

```bash
# Initialize project
breadcrumb init <project-name>

# Run with tracing
breadcrumb run -c <project> --timeout <seconds> <command>

# Smart queries
breadcrumb query -c <project> --gaps           # Find untraced calls
breadcrumb query -c <project> --call <func>    # Show function I/O
breadcrumb query -c <project> --flow           # Show execution timeline

# Inspect recorded traces
breadcrumb list -c <project>
breadcrumb get -c <project> <trace_id>
breadcrumb report -c <project>
breadcrumb top -c <project> 20
breadcrumb exceptions -c <project> --since 2h
breadcrumb clear -c <project> --force

# Config management
breadcrumb config create <name>
breadcrumb config edit <name> --add-include <pattern>
breadcrumb config edit <name> --remove-include <pattern>
breadcrumb config show <name>
breadcrumb config list
```

### Query Examples

```bash
# Find what's not traced
breadcrumb query -c myapp --gaps

# Show function with args/returns
breadcrumb query -c myapp --call process_payment

# Show execution flow for module
breadcrumb query -c myapp --flow --module myapp

# Substring call search (returns constructors too)
breadcrumb query -c myapp --call "__main__."       # All __main__ functions
breadcrumb query -c myapp --call Pizza             # Matches __main__.Pizza/__init__

# Raw SQL query (still supported)
breadcrumb query -c myapp --sql "SELECT * FROM traces WHERE status='failed'"
```

### Config Examples

```bash
# Start with minimal tracing
breadcrumb init myapp
# Creates config with include: ['__main__']

# Expand based on gaps
breadcrumb config edit myapp --add-include 'requests.*'
breadcrumb config edit myapp --add-include 'sqlalchemy.*'
breadcrumb config edit myapp --add-include 'myapp.services.*'

# Remove noisy modules
breadcrumb config edit myapp --remove-include 'logging.*'

# View current config
breadcrumb config show myapp
```

---

## Use Cases for AI Coding Agents

### 1. Understanding Legacy Code

**Human approach:**
- Read docs (if they exist)
- Step through with debugger
- Ask teammates

**AI agent approach:**
```bash
# Run the code
breadcrumb run -c legacy --timeout 120 python legacy_system.py

# See what actually executed
breadcrumb query -c legacy --flow

# Find entry points
breadcrumb query -c legacy --gaps
```

**Result:** AI agent sees actual execution flow, not just static code.

### 2. Debugging API Integration Issues

**Human approach:**
- Add print statements
- Check API logs
- Use network inspector

**AI agent approach:**
```bash
# Trace the integration
breadcrumb config edit myapp --add-include 'requests.*'
breadcrumb run -c myapp --timeout 60 python test_api.py

# See exact API calls
breadcrumb query -c myapp --call 'requests.post'
```

**Result:** AI agent sees exact request args, response values, and timing.

### 3. Performance Investigation

**Human approach:**
- Use profiler
- Add timing code
- Analyze bottlenecks

**AI agent approach:**
```bash
# Run with tracing
breadcrumb run -c myapp --timeout 300 python load_test.py

# Find slow functions
breadcrumb query -c myapp --sql "SELECT function_name, AVG(duration_ms) FROM trace_events GROUP BY function_name ORDER BY AVG(duration_ms) DESC LIMIT 10"
```

**Result:** AI agent identifies performance bottlenecks from real execution data.

### 4. Test Failure Analysis

**Human approach:**
- Re-run test
- Add debug output
- Check assertions

**AI agent approach:**
```bash
# Run failing test with tracing
breadcrumb run -c tests --timeout 60 pytest tests/test_payment.py::test_refund

# See what actually happened
breadcrumb query -c tests --call process_refund

# Check execution flow
breadcrumb query -c tests --flow --module myapp.payments
```

**Result:** AI agent sees exact execution path that led to failure.

---

## MCP Integration (Claude Desktop)

Breadcrumb includes an MCP server for Claude Desktop integration.

**Setup:**

Add to `claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "breadcrumb": {
      "command": "breadcrumb",
      "args": ["serve-mcp"],
      "env": {}
    }
  }
}
```

**Available Tools:**
- `breadcrumb__query_traces` - Execute SQL queries
- `breadcrumb__get_trace` - Get trace details by ID
- `breadcrumb__find_exceptions` - Find exceptions in traces
- `breadcrumb__analyze_performance` - Analyze function performance

**Usage:**
```
Tell Claude: "Use breadcrumb to find the last exception in my app"
Claude will use: breadcrumb__find_exceptions
```

---

## Advanced Configuration

### Python API (Optional)

If you need programmatic control, you can still use the Python API:

```python
import breadcrumb

# Basic configuration
breadcrumb.init(
    enabled=True,
    sample_rate=1.0,
    db_path="~/.breadcrumb/myapp-traces.duckdb",
    include=["myapp.*", "requests.*"]
)

# Your code here
def my_function():
    pass
```

**But 99% of use cases should use `breadcrumb run` instead.**

### Environment Variables

```bash
# Override config settings
export BREADCRUMB_ENABLED=true
export BREADCRUMB_DB_PATH="/custom/path/traces.duckdb"
export BREADCRUMB_SAMPLE_RATE=0.5
```

---

## Database Schema

Breadcrumb stores traces in DuckDB with these tables:

**traces** - One row per execution:
```sql
CREATE TABLE traces (
    id VARCHAR PRIMARY KEY,
    started_at TIMESTAMP,
    ended_at TIMESTAMP,
    status VARCHAR,  -- 'running', 'completed', 'failed'
    duration_ms DOUBLE
);
```

**trace_events** - Function calls and returns:
```sql
CREATE TABLE trace_events (
    id VARCHAR PRIMARY KEY,
    trace_id VARCHAR,
    timestamp TIMESTAMP,
    event_type VARCHAR,  -- 'call', 'return', 'exception', 'call_site'
    function_name VARCHAR,
    module_name VARCHAR,
    data JSON  -- Contains: args, kwargs, return_value, caller info
);
```

**Direct SQL access:**
```python
import duckdb
conn = duckdb.connect('~/.breadcrumb/myapp-traces.duckdb')
result = conn.execute("SELECT * FROM traces").fetchall()
```

---

## How It Works

### Architecture

1. **PEP 669 Backend (Python 3.12+)**
   - Uses sys.monitoring API (low overhead)
   - Captures function calls/returns automatically
   - Detects untraced calls for gap detection

2. **Trace Storage (DuckDB)**
   - Fast columnar database
   - JSON support for flexible data
   - SQL queryable for complex analysis

3. **Smart Query Layer**
   - Abstracts SQL complexity
   - Returns structured JSON
   - Optimized for AI agent consumption

4. **CLI Interface**
   - Typer-based commands
   - Explicit project specification
   - AI-FIRST design principles

### Include-Only Workflow

**Default behavior:** Trace only `__main__` module (minimal overhead)

**Iterative expansion:**
```bash
breadcrumb run -c myapp python script.py     # Trace __main__ only
breadcrumb query -c myapp --gaps             # Find what's not traced
breadcrumb config edit myapp --add-include 'requests.*'  # Expand
breadcrumb run -c myapp python script.py     # Trace more
```

**Philosophy:** Start minimal, expand based on what you discover.

This is **much better** than tracing everything and filtering later.

---

## Performance

Breadcrumb is designed for production use:

**Overhead benchmarks (1M function calls):**
- Baseline: 0.85s
- With Breadcrumb (PEP 669): 0.87s (2.4% overhead)
- With Breadcrumb (settrace): 0.89s (4.7% overhead)

**Optimization strategies:**
- Selective tracing (include patterns)
- Async database writes
- Value truncation (prevent bloat)
- Smart sampling (optional)

---

## Security

**Automatic secret redaction** prevents sensitive data from reaching traces:

**Detected patterns:**
- Password fields (`password`, `passwd`, `pwd`, `secret`)
- API keys (`api_key`, `apikey`, `token`, `bearer`)
- Credit cards (16-digit patterns)
- SSNs (XXX-XX-XXXX format)
- JWTs (eyJ... tokens)
- AWS keys (AKIA... patterns)
- GitHub tokens (ghp_... patterns)

**Custom patterns:**
```python
from breadcrumb.capture.secret_redactor import configure_redactor
configure_redactor(custom_patterns=['my_secret_*', 'internal_token'])
```

---

## Requirements

- **Python 3.12+** (recommended for PEP 669 backend)
- **Python 3.10+** (falls back to sys.settrace backend)
- **DuckDB 1.4.1+**
- **FastMCP 2.12.4+** (for MCP server)
- **Typer 0.19.2+** (for CLI)

---

## Status

**Current version:** 0.2.0 (Beta)

**Production-ready features:**
- ✅ PEP 669 tracing backend
- ✅ Zero-code-change execution (`breadcrumb run`)
- ✅ Gap detection (`--gaps` command)
- ✅ Smart queries (no SQL needed)
- ✅ AI-FIRST design (explicit `-c` everywhere)
- ✅ Automatic secret redaction
- ✅ DuckDB storage
- ✅ MCP server integration

**Coming soon:**
- 🚧 `--call` command (show function I/O details)
- 🚧 `--flow` command (execution timeline)
- 📦 PyPI package
- 📊 Performance visualizations
- ☁️ Cloud storage backends

---

## Contributing

We welcome contributions! This project is built for AI coding agents.

**Development setup:**
```bash
git clone https://github.com/AndreRatzenberger/breadcrumb-tracer.git
cd breadcrumb-tracer
uv pip install -e .
pytest tests/
```

**Key principles:**
- AI-FIRST design (explicit over implicit)
- Zero code changes required
- Performance matters (< 5% overhead)
- Test-driven development

---

## License

MIT License - see LICENSE file for details.

---

## Acknowledgments

Built with:
- [PEP 669 (sys.monitoring)](https://peps.python.org/pep-0669/) - Low-overhead monitoring API
- [DuckDB](https://duckdb.org/) - Embedded analytical database
- [FastMCP](https://github.com/jlowin/fastmcp) - MCP server framework
- [Typer](https://typer.tiangolo.com/) - CLI framework

Special thanks to the Model Context Protocol team at Anthropic for enabling AI-native tooling.

---

## The Bottom Line

**Humans debug with debuggers.**
**AI agents debug with execution traces.**

Breadcrumb gives AI coding agents the X-ray vision they need to understand what your code actually does - not just what it's supposed to do.

No code changes. No manual logging. No guessing.

Just structured execution data, queryable in real-time.

**That's Breadcrumb.** 🍞