topaz-agent-kit 0.12.5
Config-driven multi-agent orchestration kit with optional API/UI/MCP extras
License: MIT
Requires Python: >=3.11
Keywords: nishant, topaz, agents, multi-agent, orchestration, fastapi, pipeline

<div align="center">
  <img src="src/topaz_agent_kit/templates/foundations/basic/ui/static/assets/tak-logo.png" alt="Topaz Agent Kit Logo" width="200"/>
</div>

# Topaz Agent Kit

**A powerful, config-driven multi-agent orchestration framework for building sophisticated AI workflows across multiple frameworks and protocols.**

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## 🚀 What is Topaz Agent Kit?

Topaz Agent Kit is a **lightweight, developer-focused toolkit** that enables rapid creation, deployment, and orchestration of AI agent workflows. Instead of building from scratch, teams can assemble sophisticated multi-agent demos in hours, experiment freely across frameworks, and focus on delivering real business value.

### ✨ Key Features

- **🔄 Framework Flexibility**: Build with LangGraph, CrewAI, Agno, ADK, OAK, or MAF — mix and match without being tied to a single ecosystem
- **🔗 Protocol Choice**: Create agents that work with A2A or IN-PROC protocols for flexible multi-agent communication
- **🧰 Rich Tool Ecosystem**: Pre-built MCP toolkits for document processing, web search, math, email, SEC filings, and more
- **🖼️ Multimodal Support**: Native support for images, documents, and URLs across all frameworks — agents process visual and document inputs directly
- **⚡ Rapid Development**: From idea to working demo in hours, not weeks
- **🎛️ Modern Web UI**: Real-time agent interaction visualization with drag-and-drop file uploads, session management, AG-UI protocol, and script execution
- **📱 App Mode**: Config-driven UI apps with **three canvas modes** (Declarative, Agent, Hybrid), 47 widget types, and flexible layouts for collaborative AI-assisted editing
- **🎛️ Operations Center**: Centralized case management interface with pipeline-specific views, dashboard analytics, Operations Assistant, and comprehensive case detail modals
- **🚪 Enhanced HITL**: Multi-type gates (approval, input, selection) with **async HITL support** for batch processing, dynamic flow control (retry, skip, stop), and context injection
- **⚡ Async HITL**: Non-blocking human-in-the-loop with checkpoint management, case queuing, and pipeline resumption for high-throughput workflows
- **📄 Document Intelligence**: Complete RAG-as-a-Service with document upload, analysis, semantic search, and citation support
- **🔄 10 Execution Patterns**: Sequential, parallel, repeat, loop, conditional, switch, handoff, group chat, nested patterns, and pipeline composition
- **🔗 Pipeline Composition**: Use entire pipelines as nodes within other pipelines for maximum reusability
- **⚡ Event-Driven Pipelines**: Automatic pipeline execution triggered by file system events, webhooks, or custom triggers
- **🎨 Auto-Generated Diagrams**: Professional workflow diagrams with Graphviz for all pipelines
- **🧠 Prompt Intelligence**: Automatic variable detection and context-aware agent generation
- **💬 Intelligent Routing**: LLM-based pipeline selection based on user intent
- **📋 SOP-Driven Agents**: Standard Operating Procedure support for structured, consistent agent workflows
- **🧠 AgentOS Memory**: Filesystem-based memory system with Unix-like commands for persistent agent memory, shared templates, and semantic search
- **📦 Production Ready**: Built-in monitoring, error handling, logging, and scalability features

## 🏗️ Architecture Overview

### Multi-Framework Support

Topaz Agent Kit supports **6 major AI frameworks** with unified orchestration:

| Framework     | Description                  | Best For                        | Multimodal Support                   |
| ------------- | ---------------------------- | ------------------------------- | ------------------------------------ |
| **LangGraph** | State-based agent workflows  | Complex stateful processes      | ✅ Images, Documents, URLs           |
| **CrewAI**    | Collaborative agent teams    | Multi-agent collaboration       | ⚠️ URLs only (local file limitation) |
| **Agno**      | Lightweight agent framework  | Simple, fast agent tasks        | ✅ Images, Documents, URLs           |
| **ADK**       | Google Agent Development Kit | Google ecosystem integration    | ✅ Images, Documents, URLs           |
| **OAK**       | OpenAI Agents Kit            | OpenAI model optimization       | ✅ Images, Documents, URLs           |
| **MAF**       | Microsoft Agent Framework    | Enterprise agent orchestration  | ✅ Images, Documents, URLs           |

**Note**: Semantic Kernel (SK) support has been removed due to dependency conflicts with A2UI protocol support. All SK agents have been migrated to Microsoft Agent Framework (MAF), which is Microsoft's recommended path forward. See [Framework Limitations](#framework-limitations) for details.

**Multimodal Input Support**: All frameworks (except CrewAI with limitations) can process images, documents, and URLs directly as agent input. Files are automatically preprocessed and passed to agents in framework-native formats:

- **Images**: Local files and URLs supported (base64-encoded for most frameworks)
- **Documents**: PDFs, text files, and other document types (extracted text included when available)
- **URLs**: Automatically detected in user messages and processed as images or documents

**Note**: CrewAI has known limitations with local image processing when using Azure OpenAI models. Image URLs work reliably; local files may require alternative approaches.

### Framework Limitations

#### Semantic Kernel (SK) Removal

**Status**: ❌ **Completely Removed** - Semantic Kernel framework support has been completely removed from Topaz Agent Kit as of this version.

**Reason**: SK had a dependency conflict with A2UI protocol support:
- SK requires `pydantic<2.12`
- A2UI requires `pydantic>=2.12.5`
- These requirements are incompatible and cannot be resolved

**What Was Removed**:
- `SKBaseAgent` class and all SK framework code
- SK framework configuration (`sk.yml`)
- SK from framework validation and factory registration
- All SK-related imports and dependencies

**Migration**: All existing SK agents have been migrated to **Microsoft Agent Framework (MAF)**, which is Microsoft's recommended path forward for agent development. MAF provides similar capabilities to SK with better future support and full A2UI compatibility.

**For Existing Projects**: If you have SK agents in your project:
1. Change `type: sk` to `type: maf` in agent config files
2. Regenerate agents: `topaz-agent-kit generate <project_dir>`
3. Test your agents (MAF has similar capabilities to SK)

**For New Projects**: Use MAF, LangGraph, CrewAI, Agno, ADK, or OAK frameworks. SK is no longer available.

**Note**: The SK base class (`sk_base_agent.py`) and configuration file (`sk.yml`) have been completely removed from the codebase. Attempting to use `type: sk` in agent configurations will result in validation errors.

See [SK and A2UI Dependency Conflict Documentation](docs/dependencies/sk_a2ui_conflict.md) for detailed migration guide and technical details.

### Execution Patterns

Revolutionary **pattern-based execution** replaces complex graph definitions:

- **Sequential**: Run agents in order with dependency management
- **Parallel**: Execute multiple agents simultaneously for maximum efficiency
- **Repeat**: Execute the same agent multiple times in parallel with instance-specific inputs (new)
- **Loop**: Iterative execution with configurable termination conditions
- **Conditional**: Execute agents or entire branch based on dynamic conditions
- **Switch**: Route to different branches based on field values or expressions
- **Handoff**: LLM-driven routing to specialist agents with auto-generated orchestrator
- **Group Chat**: Multiple agents collaborate in a shared conversation (new)
- **Nested**: Combine patterns for complex workflows
- **Pipeline Composition**: Use entire pipelines as nodes within other pipelines for maximum reusability (new)

#### Pattern Containers & Grouping (UI)
- The UI renders **pattern containers** for every pattern_started/pattern_finished pair across all patterns (sequential, parallel, repeat, loop, switch, handoff, group_chat). Containers show name/description, status, timestamps, and elapsed time.
- Child items (agent cards, protocol chips, HITL gates, nested pattern containers) attach to a parent pattern via `parent_pattern_id`, preserving hierarchy even for deep nesting and repeat/loop instances.
- Concrete examples from starter pipelines:
  - **Haiku Writers Room (group_chat)**: One `group_chat` container with the three participants (`haiku_form_expert`, `haiku_imagery_specialist`, `haiku_editor`) as child cards in round order.
  - **Math Compass (sequential + switch)**: Top-level sequential container; inside it a switch container (“Complexity Router”) with a nested sequential path for the TRUE branch containing the calculator agent card. Protocol chips sit immediately before their target card inside the parent container.
  - **TCI Policy Risk Assessor (loop + nested sequential + parallel + switch)**: Loop container per application; inside it a nested sequential “Single Application Assessment Flow”; inside that two parallel containers (“Data Collection Phase”, “Risk Factor Assessment”) followed by a switch container (“Decision Routing”) whose cases each render as nested sequential containers.
  - **ECI Claims Vetter (loop + nested sequential + parallel)**: Loop over claims; each iteration shows a sequential container (“Single Claim Processing Flow”) containing a parallel container (“Parallel Validation Checks”) and the involved agent cards.
  - **Handoff patterns** (e.g., specialist routing): Handoff container with orchestrator card and the chosen specialist card inside; pattern_finished marks completion or failure.
- Backend emits `pattern_started`/`pattern_finished` with name, description, `parent_pattern_id`, started_at/ended_at; the frontend groups by these IDs and sorts by timestamp/priority so workflow → patterns → cards/chips/HITL render in chronological order. Each repeat/loop instance and nested pattern gets a unique `pattern_id`, preventing cross-iteration mixing.

#### Using Variables in Pattern Descriptions

Pattern descriptions support **Jinja2 templating** to display dynamic information about the current execution context. This is particularly useful for showing relevant data about items being processed in loops or conditional branches.

##### Key Concepts

1. **Timing**: Pattern descriptions are rendered **BEFORE** agents execute, so you can only use variables from:
   - Scanner outputs (e.g., `eci_pending_claims_scanner.pending_claims_list`)
   - Loop items (e.g., `current_claim`, `current_application`)
   - Database fields already stored
   - **Cannot use**: Extractor outputs or agent results (they haven't run yet)

2. **Variable Access**: Use the same Jinja2 syntax as in agent inputs:
   - Simple variables: `{{variable_name}}`
   - Nested access: `{{agent_id.field}}` or `{{current_item.field}}`
   - Filters: `{{amount | round(2)}}`
   - Conditionals: `{% if condition %}...{% endif %}`

3. **Table Formatting**: Use markdown tables to display structured data:
   - **Multi-column tables** for loop descriptions (showing all items)
   - **Two-column tables** (Field | Value) for single item details

##### Example: Loop Description (All Items)

Display a summary table of all items being processed:

```yaml
pattern:
  type: loop
  name: "Claims Processing Loop"
  description: |
    ## Iterative Claim Processing
    
    Processes each pending claim through the complete vetting workflow.
    
    {% if eci_pending_claims_scanner.pending_claims_list and eci_pending_claims_scanner.pending_claims_list | length > 0 %}
    ## Pending Claims Summary
    
    | Claim ID | Seller | Claimant | Claim Amount | Claim Reason |
    |----------|--------|----------|-------------|-------------|
    {% for claim in eci_pending_claims_scanner.pending_claims_list %}
    | {{ claim.claim_id }} | {{ claim.seller_name or 'N/A' }} | {{ claim.claimant_name or 'N/A' }} | {% if claim.invoice_amount %}${{ claim.invoice_amount | round(2) }}{% else %}N/A{% endif %} | {{ claim.claim_reason_category or 'N/A' }} |
    {% endfor %}
    {% endif %}
  iterate_over: "eci_pending_claims_scanner.pending_claims_list"
  loop_item_key: "current_claim"
```

**Result**: Shows a table with all pending claims before processing begins.

##### Example: Single Item Description (Two-Column Format)

Display detailed information about the current item being processed:

```yaml
pattern:
  type: sequential
  name: "Single Claim Processing Flow"
  description: |
    ## Individual Claim Vetting Process
    
    Complete workflow for processing a single claim.
    
    {% if current_claim %}
    ## Current Claim Details
    
    | Field | Value |
    |-------|-------|
    | Claim ID | {{ current_claim.claim_id }} |
    {% if current_claim.policy_number %} | Policy Number | {{ current_claim.policy_number }} |
    {% endif %}{% if current_claim.seller_name %} | Seller | {{ current_claim.seller_name }} |
    {% endif %}{% if current_claim.buyer_name %} | Buyer | {{ current_claim.buyer_name }} |
    {% endif %}{% if current_claim.invoice_amount %} | Claim Amount | ${{ current_claim.invoice_amount | round(2) }} |
    {% endif %}{% if current_claim.claim_reason_category %} | Claim Reason | {{ current_claim.claim_reason_category }} |
    {% endif %}
    {% endif %}
```

**Result**: Shows detailed information about the current claim in a two-column format.

##### Example: Application Loop

```yaml
pattern:
  type: loop
  name: "Applications Processing Loop"
  description: |
    ## Iterative Application Processing
    
    Processes each pending application through the complete risk assessment workflow.
    
    {% if tci_pending_applications_scanner.pending_applications_list and tci_pending_applications_scanner.pending_applications_list | length > 0 %}
    ## Pending Applications Summary
    
    | Application ID | Applicant (Seller) | Buyer | Requested Credit Limit | Requested Term |
    |----------------|-------------------|-------|----------------------|----------------|
    {% for app in tci_pending_applications_scanner.pending_applications_list %}
    | {{ app.application_id }} | {{ app.seller_name or 'N/A' }} | {{ app.buyer_name or 'N/A' }} | {% if app.requested_amount %}{{ app.currency_symbol or '$' }}{{ app.requested_amount | format_currency }}{% else %}N/A{% endif %} | {% if app.requested_term_days %}{{ app.requested_term_days }} days{% else %}N/A{% endif %} |
    {% endfor %}
    {% endif %}
  iterate_over: "tci_pending_applications_scanner.pending_applications_list"
  loop_item_key: "current_application"
```

##### Dynamic List Iteration (dynamic_iterate_over)

For recursive discovery or scenarios where new items are added during iteration, enable dynamic re-evaluation of the list:

```yaml
pattern:
  type: sequential
  steps:
    - node: scanner
    - type: loop
      iterate_over: "scanner.items_list"
      loop_item_key: "current_item"
      dynamic_iterate_over: true  # Re-evaluate list before each iteration
      termination:
        max_iterations: 50  # Safety limit (required for dynamic mode)
      body:
        type: sequential
        steps:
          - node: processor
          - node: discovery_agent  # May add new items to scanner.items_list
          - node: recorder
    - node: finalizer
```

**Key Features**:
- **`dynamic_iterate_over: true`**: Re-evaluates the `iterate_over` list before each iteration
- **Picks up new items**: Items added during iteration (e.g., by `discovery_agent`) are automatically processed
- **Duplicate prevention**: Tracks processed items to avoid processing the same item twice
- **Safety limit**: Always use `max_iterations` to prevent infinite loops

**How It Works**:
1. Before each iteration, the loop re-resolves the `iterate_over` path
2. Filters out already processed items (using item ID or hash)
3. Processes the first unprocessed item
4. Continues until no new items are found or `max_iterations` is reached

**Use Cases**:
- **Recursive Discovery**: Finding related items that may lead to more items (e.g., ReconVoy pipeline)
- **Database Updates**: Processing items that may trigger new items to be added to the queue
- **Dynamic Workloads**: When the work list grows during processing

**Example: Recursive Discovery**:
```yaml
- type: loop
  iterate_over: "related_items_discovery.related_items"
  loop_item_key: "related_item"
  dynamic_iterate_over: true  # Pick up newly discovered items
  termination:
    max_iterations: 50  # Safety limit
  body:
    type: sequential
    steps:
      - node: item_discovery  # Finds foreign book matches
      - node: related_items_discovery  # May add more items to related_items list
```

**Performance Note**: Dynamic iteration re-evaluates the list on each iteration, which may have performance implications for expensive operations (database queries, complex context resolution). Use static iteration (`dynamic_iterate_over: false` or omitted) when the list doesn't change during processing.

##### Accessing Accumulated Loop Results

When `accumulate_results` is enabled (default: `true`), loop patterns automatically create `*_instances` aliases for downstream agents, similar to repeat patterns. This allows downstream agents to access all loop iteration results, not just the last one.

**How it works**:
- After the loop completes, for each agent that ran inside the loop body, a dictionary is created with the pattern `{agent_id}_instances`
- Each iteration's result is stored with a key like `{agent_id}_0`, `{agent_id}_1`, etc.
- Downstream agents can access all iterations using `{{agent_id}_instances}` in their prompts or inputs

**Example**:
```yaml
pattern:
  type: sequential
  steps:
    - node: question_loader
    - type: loop
      iterate_over: "question_loader.questions_list"
      loop_item_key: "current_question"
      accumulate_results: true  # Default, but explicit for clarity
      body:
        type: sequential
        steps:
          - node: validator  # Runs for each question
    - node: summary_reporter  # Can access all validator results
```

In the `summary_reporter` agent configuration, you can access all validation results:
```yaml
# config/agents/summary_reporter.yml
prompt:
  inputs:
    inline: |
      - Total Questions: {{question_loader.total_count}}
      - Validation Results: {{validator_instances}}
```

The `validator_instances` dictionary will contain:
```python
{
  "validator_0": {"validation_results": {...}, "tools_used": {...}},
  "validator_1": {"validation_results": {...}, "tools_used": {...}},
  "validator_2": {"validation_results": {...}, "tools_used": {...}}
}
```

**Key Points**:
- Only agents that run **inside** the loop body get `*_instances` aliases
- The alias is created automatically when `accumulate_results=true` (default)
- Instance keys follow the pattern `{agent_id}_{iteration_index}` (0-based)
- Prefers structured `parsed` output when available, otherwise uses raw result
- This mirrors repeat pattern behavior for consistency across patterns

**When to use `accumulate_results`**:
- ✅ **Enable** (default): When you need downstream agents to process all loop results (e.g., summary reports, aggregations, batch processing)
- ❌ **Disable**: When you only need the final iteration's result or want to reduce memory usage for very large loops

##### Best Practices

1. **Use Conditional Rendering**: Always check if variables exist before using them:
   ```jinja
   {% if current_claim.seller_name %} | Seller | {{ current_claim.seller_name }} |{% endif %}
   ```

2. **Provide Fallbacks**: Use `or 'N/A'` for optional fields:
   ```jinja
   {{ claim.seller_name or 'N/A' }}
   ```

3. **Format Numbers**: Use built-in Jinja2 filters for currency and decimals:
   ```jinja
   {{ amount | format_currency }}  # "125,000.00"
   {{ 0.85 | format_percentage }}  # "85.0%"
   {{ amount | format_number(decimals=2) }}  # "125,000.00"
   ```
   
   See [Available Jinja2 Filters](#available-jinja2-filters) for all formatting options.

4. **Loop Descriptions**: Show all items in a multi-column table
5. **Single Item Descriptions**: Use two-column format (Field | Value) for readability

6. **Scanner Data**: Ensure your scanner agents extract all needed fields from the database:
   ```sql
   SELECT claim_id, seller_name, claimant_name, invoice_amount, claim_reason_category
   FROM claims WHERE status = 'pending'
   ```

7. **Table Formatting**: Use Jinja2 whitespace control to ensure proper markdown table formatting:
   - Use `{%-` and `-%}` to strip whitespace around tags
   - Separate conditional rows onto their own lines
   - Example:
   ```jinja
   | Field | Value |
   |-------|-------|
   | Claim ID | {{ current_claim.claim_id }} |
   {%- if current_claim.seller_name %}
   | Seller | {{ current_claim.seller_name }} |
   {%- endif %}
   {%- if current_claim.buyer_name %}
   | Buyer | {{ current_claim.buyer_name }} |
   {%- endif %}
   ```
   
   For loop tables:
   ```jinja
   | Claim ID | Seller | Claimant |
   |----------|--------|----------|
   {%- for claim in scanner.pending_claims_list %}
   | {{ claim.claim_id }} | {{ claim.seller_name }} | {{ claim.claimant_name }} |
   {%- endfor %}
   ```
   
   **Why**: Jinja2 whitespace control (`{%-` and `-%}`) prevents extra blank lines that break markdown table formatting.

##### Markdown Table Formatting Best Practices

Based on testing and validation, follow these rules to ensure tables render correctly:

**Critical Rules:**

1. **No Blank Lines Within Tables**: Blank lines between table rows break markdown table formatting
   ```jinja
   ❌ BAD:
   {% if condition %}
   
   | Status | {{ status }} |
   {% endif %}
   
   ✅ GOOD:
   {%- if condition %}
   | Status | {{ status }} |
   {%- endif %}
   ```

2. **Whitespace Control for Conditionals**: Always use `{%-` and `-%}` for conditionals in table cells
   ```jinja
   ❌ BAD:
   | Amount | {% if sym %}{{ sym }}{{ amount | format_currency }}{% else %}N/A{% endif %} |
   
   ✅ GOOD:
   | Amount | {%- if sym %}{{ sym }}{{ amount | format_currency }}{%- else %}N/A{%- endif %} |
   ```

3. **Nested Conditionals**: Apply whitespace control to conditionals, but NOT to `{% set %}` statements (they need newlines for markdown tables)
   ```jinja
   ❌ BAD (whitespace control on set strips newline):
   | Amount | {%- if has_amount %}{%- set sym = currency_symbol %}{%- if sym | length >= 3 %}{{ sym }} {{ amount }}{%- else %}{{ sym }}{{ amount }}{%- endif %}{%- else %}N/A{%- endif %} |
   
   ✅ GOOD (no whitespace control on set, preserves newlines):
   | Amount | {%- if has_amount %}{% set sym = currency_symbol %}{%- if sym | length >= 3 %}{{ sym }} {{ amount }}{%- else %}{{ sym }}{{ amount }}{%- endif %}{%- else %}N/A{%- endif %} |
   ```

4. **checks_table Pattern**: Use `{{ checks_table }}` (not `{{- checks_table -}}`) to preserve newlines, and ensure blank line after `</summary>`. Use `{% if %}` (NOT `{%- if %}`) for conditionals around checks_table to preserve newlines.
   ```jinja
   ❌ BAD (strips newlines, breaks table):
   {%- if agent.checks_table %}{{- agent.checks_table -}}{%- else %}
   **Summary:** {{ agent.details }}
   {%- endif %}
   
   ❌ BAD (no blank line after </summary>, table won't render):
   <details>
   <summary>Test Summary</summary>
   {%- if agent.checks_table %}
   {{ agent.checks_table }}
   {%- endif %}
   </details>
   
   ✅ GOOD (preserves newlines, blank line after </summary>, no whitespace control on if):
   <details>
   <summary>Test Summary</summary>
   
   {% if agent.checks_table %}
   {{ agent.checks_table }}
   
   {% endif %}
   </details>
   ```

5. **Set Statements Before Table Rows**: Use `{% set %}` (NOT `{%- set %}`) when used before markdown tables, especially in `<details>` blocks. The whitespace control strips newlines that markdown tables need.
   ```jinja
   ❌ BAD (whitespace control strips newline, breaks table rendering):
   <details>
   <summary>Test Summary</summary>
   
   {%- set fh_score = assessor.score if assessor else None %}
   {% if agent.checks_table %}
   {{ agent.checks_table }}
   {% endif %}
   </details>
   
   ✅ GOOD (no whitespace control on set, preserves newlines):
   <details>
   <summary>Test Summary</summary>
   
   {% set fh_score = assessor.score if assessor else None %}
   {% if agent.checks_table %}
   {{ agent.checks_table }}
   {% endif %}
   </details>
   ```
   
   **Also**: Put `{% set %}` statements on separate lines before table rows, not inline
   ```jinja
   ❌ BAD (set on same line as table row, breaks formatting):
   | Risk Factor | Score | Weight | Weighted Score |
   |-------------|-------|--------|----------------|
   {% set fh_score_val = tci_financial_health_assessor.score if tci_financial_health_assessor else None %}| Financial Health | {{ fh_score_val | default_if_none }} | 0.22 | {{ fh_weighted | default_if_none }} |
   
   ✅ GOOD (set on separate line before table row):
   | Risk Factor | Score | Weight | Weighted Score |
   |-------------|-------|--------|----------------|
   {% set fh_score_val = tci_financial_health_assessor.score if tci_financial_health_assessor else None %}
   {% set fh_weighted = (fh_score_val * 0.22) | round(2) if fh_score_val is not none else None %}
   | Financial Health | {{ fh_score_val | default_if_none }} | 0.22 | {{ fh_weighted | default_if_none }} |
   ```

6. **Filters Don't Introduce Whitespace**: Filters themselves are safe, but Jinja2 tags need control
   ```jinja
   ✅ SAFE (filters don't add whitespace):
   | Amount | {{ amount | format_currency }} |
   | Status | {{ status | default_if_none }} |
   
   ⚠️ NEEDS CONTROL (tags add whitespace):
   | Amount | {%- set sym = currency_symbol %}{{ amount | format_currency(currency_symbol=sym) }} |
   ```

7. **Tables Inside HTML Tags**: Markdown tables inside HTML tags (like `<details>`) require a blank line after the closing tag
   ```jinja
   ❌ BAD (table starts on same line as </summary>):
   <details>
   <summary>Test Summary</summary>
   | Check | Status |
   |-------|--------|
   | Test | PASS |
   </details>
   
   ✅ GOOD (blank line after </summary>):
   <details>
   <summary>Test Summary</summary>
   
   | Check | Status |
   |-------|--------|
   | Test | PASS |
   </details>
   ```

**Key Learnings from Testing**:
- **Blank lines within tables**: Break markdown rendering - never include blank lines between table rows
- **Blank lines before tables**: Required when tables are inside HTML tags (e.g., `<details>`)
- **checks_table pattern**: Use `{{ checks_table }}` (not `{{- checks_table -}}`) to preserve newlines that are part of the table structure
- **Whitespace control for conditionals**: Use `{%-` and `-%}` for conditionals and loops within table cells
- **Set statements**: Use `{% set %}` (NOT `{%- set %}`) when used before markdown tables - whitespace control strips newlines that markdown parsers need
- **Set statement placement**: Place `{% set %}` statements on separate lines before table rows, not inline

**Testing**: Use the `tests/test_markdown_tables.py` script to validate table formatting:
```bash
python tests/test_markdown_tables.py
# or
python -m pytest tests/test_markdown_tables.py -v
```

The test script generates markdown files in `tests/output/markdown_tables/` for visual inspection and validates:
- No blank lines within tables
- Tables don't start on the same line as previous content
- Consistent column counts across all table rows
- Proper whitespace control in conditionals

**Testing**: Use the `tests/test_markdown_tables.py` script to validate table formatting before deployment:
```bash
python tests/test_markdown_tables.py
# or
python -m pytest tests/test_markdown_tables.py -v
```

##### Available Jinja2 Filters

The Topaz Agent Kit provides a comprehensive set of Jinja2 filters for formatting and styling in templates. These filters are automatically available in:
- Pattern descriptions (pipeline YAML files)
- HITL gate descriptions
- Agent input templates (YAML `inputs.inline` sections)

**Number Formatting:**
```jinja
{{ 125000 | format_currency }}           # "125,000.00"
{{ 125000 | format_currency(decimals=0) }}  # "125,000"
{{ 1250.5 | format_number(decimals=2) }}    # "1,250.50"
{{ 0.85 | format_percentage }}             # "85.0%"
{{ 0.8523 | format_percentage(decimals=2) }} # "85.23%"
```

**Score/Risk Color Coding:**
```jinja
{# For risk scores where lower is better (0-25 green, 26-50 amber, etc.) #}
<span style="color: {{ risk_score | risk_score_color }};">{{ risk_score }}</span>

{# For credit/quality scores where higher is better (85-100 green, 70-84 amber, etc.) #}
<span style="color: {{ credit_score | credit_score_color }};">{{ credit_score }}</span>

{# Generic score color with custom thresholds #}
<span style="color: {{ score | score_color(thresholds=[(80, "#green"), (60, "#yellow"), (0, "#red")]) }};">{{ score }}</span>
```

**Text Formatting:**
```jinja
{{ long_text | truncate_text(50) }}              # Truncate to 50 chars with "..."
{{ 5 | pluralize("item") }}                      # "items"
{{ 1 | pluralize("child", "children") }}         # "child"
{{ text | highlight_text("search term") }}        # Highlight search terms
```

**Date/Time Formatting:**
```jinja
{{ "2025-01-28" | format_date }}                    # "2025-01-28"
{{ "2025-01-28" | format_date("%B %d, %Y") }}       # "January 28, 2025"
{{ 3665 | format_duration }}                        # "1 hour 1 minute 5 seconds"
{{ 3665 | format_duration(compact=True) }}          # "1h 1m 5s"
```

**Data Formatting:**
```jinja
{{ 1572864 | format_file_size }}                    # "1.5 MB"
{{ "1234567890" | mask_sensitive(4) }}              # "1234******"
{{ "1234567890" | format_phone }}                   # "(123) 456-7890"
```

**Utility Functions:**
```jinja
{{ 10 | safe_divide(2) }}                          # 5.0
{{ 10 | safe_divide(0, "N/A") }}                    # "N/A"
{{ None | default_if_none("—") }}                   # "—"
```

**Complete Filter Reference:**

| Filter | Description | Example |
|--------|-------------|---------|
| `format_currency(value, decimals=2)` | Format as currency with commas | `125000` → `"125,000.00"` |
| `format_number(value, decimals=0, thousands_sep=",")` | Format number with optional decimals | `1250.5` → `"1,250.50"` |
| `format_percentage(value, decimals=1, multiply=True)` | Format as percentage | `0.85` → `"85.0%"` |
| `risk_score_color(value)` | Color for risk scores (lower is better) | `15` → `"#22c55e"` (green) |
| `credit_score_color(value)` | Color for credit scores (higher is better) | `90` → `"#22c55e"` (green) |
| `score_color(value, thresholds, low_is_better)` | Generic score color with custom thresholds | See examples above |
| `truncate_text(value, max_length=100, suffix="...")` | Truncate text with suffix | `"Very long text"` → `"Very long..."` |
| `pluralize(value, singular, plural=None)` | Singular/plural form | `5` → `"items"` |
| `format_date(value, format_str="%Y-%m-%d")` | Format date/datetime | `"2025-01-28"` → `"January 28, 2025"` |
| `format_duration(seconds, compact=False)` | Format duration | `3665` → `"1 hour 1 minute 5 seconds"` |
| `format_file_size(value, binary=False)` | Format bytes as file size | `1572864` → `"1.5 MB"` |
| `mask_sensitive(value, visible_chars=4, mask_char="*")` | Mask sensitive data | `"1234567890"` → `"1234******"` |
| `format_phone(value, format_str="us")` | Format phone number | `"1234567890"` → `"(123) 456-7890"` |
| `safe_divide(numerator, denominator, default=0)` | Safe division | `10 / 0` → `0` |
| `default_if_none(value, default="N/A")` | Default for None values | `None` → `"N/A"` |

##### Available Variables by Context

- **Loop Description**: Access the full list from scanner (e.g., `scanner.pending_list`)
- **Loop Body (Sequential/Parallel)**: Access current item via `loop_item_key` (e.g., `current_claim`, `current_application`)
- **Conditional/Switch Branches**: Access variables from upstream agents or context
- **Not Available**: Extractor outputs or downstream agent results (they haven't executed yet)

##### Hierarchy snapshots (cards/chips/patterns)
- **Haiku Writers Room (group_chat)**  
  - group_chat (Haiku Writers Room)  
    - card: haiku_form_expert  
    - card: haiku_imagery_specialist  
    - card: haiku_editor  

- **Math Compass (sequential + switch)**  
  - sequential: Math Problem Solving Flow  
    - chip: orchestrator → math_strategist  
    - card: math_strategist  
    - switch: Complexity Router  
      - sequential (TRUE case)  
        - chip: math_strategist → math_calculator  
        - card: math_calculator  

- **TCI Policy Risk Assessor (loop → sequential → parallel → switch)**  
  - loop: Applications Processing Loop (one per application)  
    - sequential: Single Application Assessment Flow  
      - parallel: Data Collection Phase (collectors)  
      - parallel: Risk Factor Assessment (risk scorers)  
      - switch: Decision Routing  
        - sequential: case Modified Terms  
        - sequential: case Rejection  
        - sequential: case Escalation  
        - sequential: case Information Request  

- **ECI Claims Vetter (loop → sequential → parallel)**  
  - loop: Claims Processing Loop (one per claim)  
    - sequential: Single Claim Processing Flow  
      - parallel: Parallel Validation Checks (e.g., history lookup, eligibility)  

- **Handoff**  
  - handoff: <pattern name>  
    - card: orchestrator  
    - card: selected specialist  

### Protocol Support

Flexible agent communication protocols:

- **A2A** (Agent-to-Agent): Direct agent interaction patterns via SDK (default for remote agents)
- **IN-PROC**: Local in-process execution for performance (automatic for local agents)
- **Mixed**: Use different protocols per agent in the same workflow
- **Auto-detection**: Local agents automatically use IN-PROC regardless of pattern protocol

#### Why A2A Only?

**ACP (Agent Communication Protocol) support has been removed** as of version 0.3.0. This change was made for the following reasons:

1. **Official Deprecation**: The ACP SDK has been officially deprecated and merged with A2A under the Linux Foundation (August 2025). The ACP SDK is no longer actively maintained and new features are not being added.

2. **Simplified Configuration**: Removing ACP support significantly simplifies agent configuration:
   - Single `url` field instead of `urls.a2a` and `urls.acp` objects
   - No `protocol_support` array needed
   - Cleaner, more maintainable YAML files
   - Reduced port management complexity

3. **Unified Protocol**: A2A protocol now supports all ACP functionality plus additional features like path-based routing, making it the single recommended protocol for remote agent communication.

4. **Future-Proof**: A2A is actively developed with community support and is the long-term standard for agent-to-agent communication.

**Migration**: If you have existing agents using ACP, they should be migrated to A2A. The A2A SDK provides full compatibility with ACP features, and migration is straightforward - simply update the `remote.url` field in your agent configurations.

## 🧰 Built-in MCP Toolkits

Topaz Agent Kit includes **15 comprehensive MCP toolkits** with **75+ tools** ready to use:

### 📄 Document Intelligence

#### **DocExtract Toolkit** (`doc_extract_*`)
Extract structured data, tables, and metadata from documents.

- **`doc_extract_structured_data`**: Extract structured data from PDFs, DOCX, PPTX, HTML, Markdown using AI-powered field extraction
- **`doc_extract_tables`**: Extract tables from documents with structure preservation
- **`doc_extract_metadata`**: Extract document metadata (title, author, creation date, etc.)

#### **DocRAG Toolkit** (`doc_rag_*`)
Document retrieval and semantic search with ChromaDB.

- **`doc_rag_query_document`**: Query documents using semantic search with citation support
- **`doc_rag_list_documents`**: List all documents in the collection

#### **ImageRAG Toolkit** (`image_rag_*`)
Image analysis and OCR-based search.

- **`image_rag_query_images`**: Query images using OCR text content with semantic search
- **`image_rag_list_images`**: List all images in the collection

### 🌐 Web & Search

#### **Browser Toolkit** (`browser_*`)
Web scraping and automation.

- **`browser_scrape_website_content`**: Scrape and extract content from websites

#### **Serper API Toolkit** (`serper_api_*`)
Web search integration via Serper API.

- **`serper_api_search_internet`**: Search the internet with Serper API
- **`serper_api_search_news`**: Search news articles with Serper API

#### **SEC API Toolkit** (`sec_api_*`)
SEC filings and financial document retrieval.

- **`sec_api_search_10q`**: Search and retrieve SEC 10-Q quarterly reports
- **`sec_api_search_10k`**: Search and retrieve SEC 10-K annual reports

### 🔢 Math & Analysis

#### **Math Toolkit** (`math_*`)
Advanced mathematical operations and problem-solving.

**Basic Operations:**
- **`math_multiply`**: Multiply two numbers
- **`math_evaluate_expression`**: Evaluate mathematical expressions safely
- **`math_percentage_of`**: Calculate percentage of a value
- **`math_percent_change`**: Calculate percent change between two values
- **`math_fraction_to_decimal`**: Convert fraction to decimal
- **`math_ceil_divide`**: Ceiling division (round up)

**Advanced Math:**
- **`math_solve_equations`**: Solve systems of equations
- **`math_solve_inequality`**: Solve inequalities
- **`math_compute_log`**: Compute logarithms (natural or custom base)
- **`math_compute_exp`**: Compute exponential functions
- **`math_trig_functions`**: Compute trigonometric functions (sin, cos, tan, etc.)
- **`math_differentiate`**: Differentiate mathematical expressions
- **`math_integrate_expr`**: Integrate mathematical expressions

**Text Processing:**
- **`math_summarize`**: Summarize text (rule-based)
- **`math_llm_summarize`**: Summarize text using LLM
- **`math_extract_quantities`**: Extract numerical quantities from text
- **`math_llm_parse_math_problem`**: Parse math problems using LLM
- **`math_sanitize_expression`**: Sanitize mathematical expressions for safe evaluation

### 📧 Communication

#### **Email Toolkit** (`email_*`)
Gmail integration for email management via SimpleGmail.

**Email Operations:**
- **`email_send`**: Send emails with HTML/plain text, attachments, and custom headers
- **`email_list_messages`**: List messages from a label/folder with pagination
- **`email_read_message`**: Read full message content with attachments
- **`email_search`**: Search emails by query with filters
- **`email_move`**: Move emails between labels/folders
- **`email_mark_read`**: Mark emails as read
- **`email_mark_unread`**: Mark emails as unread
- **`email_delete`**: Delete emails
- **`email_download_attachment`**: Download email attachments to local path

**Email Management:**
- **`email_list_labels`**: List all Gmail labels/folders

**Helper Tools:**
- **`email_get_company_info`**: Retrieve company information
- **`email_get_email_signature`**: Generate professional email signature
- **`email_spell_check`**: Perform spell check on text

### 🗄️ Database

#### **SQLite Toolkit** (`sqlite_*`)
SQLite database operations.

- **`sqlite_query`**: Execute SELECT queries and return results as JSON
- **`sqlite_execute`**: Execute INSERT, UPDATE, DELETE statements
- **`sqlite_schema`**: Get table schema information

### 📁 Filesystem

#### **Filesystem Toolkit** (`fs_*`)
Basic file and directory operations.

-