madsci.resource_manager 0.6.1
The Modular Autonomous Discovery for Science (MADSci) Resource Manager.
License: MIT
Requires Python: >=3.9.1

# MADSci Resource Manager

Tracks and manages the full lifecycle of laboratory resources - assets, consumables, samples, containers, and labware.

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Usage](#usage)
  - [Quick Start](#quick-start)
  - [Manager Setup](#manager-setup)
- [Configuration](#configuration)
  - [Environment Variables](#environment-variables)
  - [Local Mode Configuration](#local-mode-configuration)
  - [Production Configuration](#production-configuration)
  - [Database Setup](#database-setup)
  - [Configuration Validation](#configuration-validation)
- [Resource Client](#resource-client)
- [Resource Types](#resource-types)
  - [Core Resource Hierarchy](#core-resource-hierarchy)
  - [Usage Examples](#usage-examples)
- [Integration with MADSci Ecosystem](#integration-with-madsci-ecosystem)
- [Advanced Operations](#advanced-operations)
  - [Resource Definitions](#resource-definitions)
  - [Bulk Operations](#bulk-operations)
  - [History and Auditing](#history-and-auditing)
- [Resource Templates](#resource-templates)
  - [Creating Templates](#creating-templates)
  - [Using Templates to Create Resources](#using-templates-to-create-resources)
  - [Template Management Operations](#template-management-operations)
  - [Template Use Cases](#template-use-cases)
  - [Template Best Practices](#template-best-practices)
- [Resource Locking and Concurrency Control](#resource-locking-and-concurrency-control)
  - [Basic Resource Locking](#basic-resource-locking)
  - [Context Manager for Automatic Lock Management](#context-manager-for-automatic-lock-management)
  - [Advanced Locking Patterns](#advanced-locking-patterns)
  - [Error Handling and Lock Recovery](#error-handling-and-lock-recovery)
  - [Best Practices for Resource Locking](#best-practices-for-resource-locking)
  - [Integration with Node Actions](#integration-with-node-actions)
- [Resource Hierarchy Queries](#resource-hierarchy-queries)
  - [Understanding Resource Hierarchy](#understanding-resource-hierarchy)
  - [Querying Resource Hierarchy](#querying-resource-hierarchy)
  - [Hierarchy Query Results](#hierarchy-query-results)
  - [Use Cases](#use-cases)
  - [Performance Considerations](#performance-considerations)
- [Performance Optimization](#performance-optimization)
  - [Database Performance](#database-performance)
  - [Resource Management Performance](#resource-management-performance)
  - [Lock Management Performance](#lock-management-performance)
  - [Memory Management](#memory-management)
  - [Monitoring Performance](#monitoring-performance)
  - [Performance Best Practices](#performance-best-practices)
- [Error Handling and Troubleshooting](#error-handling-and-troubleshooting)
  - [Common Error Scenarios](#common-error-scenarios)
  - [Database Connection Issues](#database-connection-issues)
  - [Performance Troubleshooting](#performance-troubleshooting)
  - [Resource State Issues](#resource-state-issues)
  - [Docker and Service Issues](#docker-and-service-issues)
  - [Common Solutions](#common-solutions)
- [Quick Reference](#quick-reference)
  - [Essential Operations](#essential-operations)
  - [Common Resource Types](#common-resource-types)
  - [Key Environment Variables](#key-environment-variables)
  - [Common Commands](#common-commands)

## Features

- **Comprehensive resource types**: Assets, consumables, containers with specialized behaviors
- **Complete history tracking**: Full audit trail with restore capabilities
- **Container hierarchy**: Supports racks, plates, stacks, queues, grids, and custom containers
- **Quantity management**: Track consumable quantities with capacity limits
- **Query system**: Find resources by type, name, properties, or relationships
- **Constraint validation**: Prevents logical errors like negative quantities or overflow

## Installation

See the main [README](../../README.md#installation) for installation options. This package is available as:

- PyPI: `pip install madsci.resource_manager`
- Docker: Included in `ghcr.io/ad-sdl/madsci`
- **Example configuration**: See [example_lab/managers/example_resource.manager.yaml](../../example_lab/managers/example_resource.manager.yaml)

**Dependencies**: PostgreSQL database (see [example_lab](../../example_lab/))

## Usage

### Quick Start

Use the [example_lab](../../example_lab/) as a starting point:

```bash
# Start with working example
docker compose up  # From repo root
# Resource Manager available at http://localhost:8003/docs

# Or run standalone
python -m madsci.resource_manager.resource_server
```

### Manager Setup

For custom deployments, see [example_resource.manager.yaml](../../example_lab/managers/example_resource.manager.yaml) for configuration options.

## Configuration

The Resource Manager uses environment variables for configuration with a hierarchical precedence system. All settings have defaults suitable for development.

### Environment Variables

**Core Settings:**
```bash
# Service Configuration
RESOURCE_HOST=localhost                    # Server hostname
RESOURCE_PORT=8003                        # Server port
RESOURCE_LOG_LEVEL=INFO                   # Logging level

# Database Configuration
RESOURCE_POSTGRES_HOST=localhost          # PostgreSQL hostname
RESOURCE_POSTGRES_PORT=5432              # PostgreSQL port
RESOURCE_POSTGRES_USER=madsci            # Database username
RESOURCE_POSTGRES_PASSWORD=madsci        # Database password
RESOURCE_POSTGRES_DATABASE=madsci        # Database name

# Manager Integration
RESOURCE_EVENT_MANAGER_URL=http://localhost:8001    # Event logging
```

**Advanced Settings:**
```bash
# Development/Testing
RESOURCE_LOCAL_MODE=false                 # Run without external dependencies
RESOURCE_ENABLE_CORS=true                # Enable CORS for web clients

# Performance Tuning
RESOURCE_MAX_CONNECTIONS=20              # Database connection pool size
RESOURCE_QUERY_TIMEOUT=30                # Query timeout in seconds
RESOURCE_LOCK_DEFAULT_DURATION=300       # Default lock duration (seconds)

# Security
RESOURCE_REQUIRE_AUTHENTICATION=false    # Enable authentication
RESOURCE_API_KEY_HEADER=X-API-Key        # API key header name
```

### Local Mode Configuration

For development or testing without external dependencies:

```bash
export RESOURCE_LOCAL_MODE=true
export RESOURCE_EVENT_MANAGER_URL=""     # Disable event logging
python -m madsci.resource_manager.resource_server
```

**Local Mode Limitations:**
- No event logging integration
- No distributed locking coordination
- Single-process operation only
- In-memory or file-based storage options

**When to Use Local Mode:**
- Unit testing and development
- Offline development environments
- Single-process applications
- Quick prototyping and experimentation

**When to Use Server Mode:**
- Production deployments
- Multi-process/multi-node environments
- Integration with other MADSci managers
- Distributed laboratory setups
- When you need event logging and audit trails

### Production Configuration

**Docker Compose (Recommended):**
```yaml
version: '3.8'
services:
  resource_manager:
    image: ghcr.io/ad-sdl/madsci:latest
    environment:
      RESOURCE_POSTGRES_HOST: postgres
      RESOURCE_POSTGRES_PASSWORD: ${DB_PASSWORD}
      RESOURCE_LOG_LEVEL: WARNING
      RESOURCE_MAX_CONNECTIONS: 50
    depends_on:
      - postgres
    ports:
      - "8003:8003"
```

**Environment File (.env):**
```bash
# Database credentials
DB_PASSWORD=secure_password_here
POSTGRES_PASSWORD=secure_password_here

# Resource Manager settings
RESOURCE_LOG_LEVEL=INFO
RESOURCE_REQUIRE_AUTHENTICATION=true
RESOURCE_API_KEY_HEADER=Authorization
```

### Database Setup

**Initial Setup:**
```bash
# Using Docker Compose
docker compose up -d postgres
docker compose exec postgres psql -U madsci -d madsci -c "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"

# Manual PostgreSQL setup
createdb -U postgres madsci
psql -U postgres -d madsci -c "CREATE USER madsci WITH PASSWORD 'madsci';"
psql -U postgres -d madsci -c "GRANT ALL PRIVILEGES ON DATABASE madsci TO madsci;"
```

**Schema Migration:**
```python
# The Resource Manager automatically creates tables on startup
from madsci.resource_manager.resource_server import ResourceManagerServer
from madsci.resource_manager.resource_server import ResourceManagerSettings

settings = ResourceManagerSettings()
server = ResourceManagerServer(settings)
# Tables created automatically when server starts
```

### Configuration Validation

```python
# Validate configuration before starting
from madsci.resource_manager.resource_server import ResourceManagerSettings

try:
    settings = ResourceManagerSettings()
    print(f"✓ Configuration valid")
    print(f"  Database: {settings.postgres_host}:{settings.postgres_port}")
    print(f"  Server: {settings.host}:{settings.port}")
except Exception as e:
    print(f"✗ Configuration error: {e}")
```

### Resource Client

Use `ResourceClient` to manage laboratory resources:

```python
from madsci.client.resource_client import ResourceClient
from madsci.common.types.resource_types import Asset, Consumable, Grid
from madsci.common.types.resource_types.definitions import ResourceDefinition

client = ResourceClient("http://localhost:8003")

# Add a new asset (samples, labware, equipment)
sample = Asset(
    resource_name="Sample A1",
    resource_class="sample",
    attributes={"compound": "aspirin", "concentration": "10mM"}
)
added_sample = client.add_resource(sample)

# Add consumables with quantities
reagent = Consumable(
    resource_name="PBS Buffer",
    resource_class="reagent",
    quantity=500.0,
    attributes={"units": "mL"},
)
added_reagent = client.add_resource(reagent)

# Create containers (plates, racks, etc.)
plate = Grid(
    resource_name="96-well Plate #1",
    resource_class="plate",
    rows=8,
    columns=12
)
added_plate = client.add_resource(plate)

# Place samples in containers
client.set_child(resource=added_plate, key=(0, 0), child=added_sample)

# Query resources
samples = client.query_resource(resource_class="sample", multiple=True)
consumables = client.query_resource(resource_class="reagent", multiple=True)

# Manage consumable quantities
client.decrease_quantity(resource=added_reagent, amount=50.0)  # Use 50mL
client.increase_quantity(resource=added_reagent, amount=100.0) # Add 100mL

# Resource history and restoration
history = client.query_history(resource_id=added_sample.resource_id)
client.remove_resource(resource_id=added_sample.resource_id)  # Soft delete
client.restore_deleted_resource(resource_id=added_sample.resource_id)

# Query resource hierarchy
hierarchy = client.query_resource_hierarchy(resource_id=added_plate.resource_id)
print(f"Ancestors: {hierarchy.ancestor_ids}")
print(f"Descendants: {hierarchy.descendant_ids}")
```

## Resource Types

### Core Resource Hierarchy

**Base Types:**
- **Resource**: Base class for all resources
- **Asset**: Non-consumable resources (samples, labware, equipment)
- **Consumable**: Resources with quantities that can be consumed

**Container Types:**
- **Container**: Base for resources that hold other resources
- **Collection**: Supports random access by key
- **Row**: Single-dimensional containers
- **Grid**: Two-dimensional containers (plates, racks)
- **VoxelGrid**: Three-dimensional containers
- **Slot**: Holds exactly zero or one child (plate nests)
- **Stack**: LIFO access (stacked plates)
- **Queue**: FIFO access (sample queues)
- **Pool**: Mixed/collocated consumables

### Usage Examples

```python
# Different container types
tip_box = Grid(resource_name="Tip Box", rows=8, columns=12, resource_class="tips")
plate_stack = Stack(resource_name="Plate Stack", resource_class="plate_storage")
sample_rack = Row(resource_name="Sample Rack", length=24, resource_class="rack")

# Container operations
client.set_child(resource=tip_box, key=(0, 0), child=tip_sample)    # Grid access
client.push(resource=plate_stack, child=new_plate)                  # Stack push
client.pop(resource=plate_stack)                                    # Stack pop
client.set_child(resource=sample_rack, key=5, child=sample)         # Row access
```

## Integration with MADSci Ecosystem

Resources integrate seamlessly with other MADSci components:

- **Workflows**: Reference resources in step locations and arguments
- **Nodes**: Access resource information during actions
- **Data Manager**: Link data to specific resources and samples
- **Event Manager**: Track resource lifecycle events

```python
# Example: Node action using resources
@action
def process_sample(self, sample_resource_id: str) -> ActionResult:
    # Get sample attributes from Resource Manager
    sample = self.resource_client.get_resource(sample_resource_id)

    # Process based on sample properties
    result = self.device.analyze(sample.attributes["compound"])

    # Update sample with results
    sample.attributes["analysis_result"] = result
    self.resource_client.update_resource(sample)

    return ActionSucceeded(data=result)
```

## Advanced Operations

### Resource Definitions
Use `ResourceDefinition` for idempotent resource creation:

```python
from madsci.common.types.resource_types.definitions import ResourceDefinition

# Creates new resource or attaches to existing one
resource_def = ResourceDefinition(
    resource_name="Standard Buffer",
    resource_class="reagent"
)
resource = client.init_resource(resource_def)  # Idempotent
```

### Bulk Operations
```python
# Query multiple resources
all_samples = client.query_resource(resource_class="sample", multiple=True)
empty_containers = client.query_resource(is_empty=True, multiple=True)

# Batch operations for consumables
for reagent in reagents:
    client.decrease_quantity(resource=reagent, amount=usage_amounts[reagent.resource_id])
```

### History and Auditing
```python
# Full resource history
history = client.query_history(resource_id=sample.resource_id)

# Query by time range and change type
import datetime
recent_updates = client.query_history(
    start_date=datetime.datetime.now() - datetime.timedelta(days=7),
    change_type="Updated"
)
```

## Resource Templates

ResourceTemplates provide reusable blueprints for creating standardized laboratory resources. Templates help ensure consistency across resource creation and reduce configuration errors.

### Creating Templates

Templates are created from existing resources and can be customized with metadata:

```python
from madsci.client.resource_client import ResourceClient
from madsci.common.types.resource_types import Grid, Consumable

client = ResourceClient("http://localhost:8003")

# Create a standard 96-well plate resource
standard_plate = Grid(
    resource_name="Standard 96-Well Plate",
    resource_class="plate",
    rows=8,
    columns=12,
    attributes={
        "well_volume": 200,  # µL
        "material": "polystyrene",
        "sterilized": True
    }
)

# Create template from the resource
plate_template = client.create_template(
    resource=standard_plate,
    template_name="standard_96_well_plate",
    description="Standard 96-well polystyrene plate for assays",
    required_overrides=["resource_name"],  # Must be customized when using
    tags=["plate", "96-well", "assay", "standard"],
    created_by="lab_manager",
    version="1.0.0"
)
```

### Using Templates to Create Resources

Templates streamline resource creation with consistent defaults:

```python
# Create new resources from template
assay_plate_1 = client.create_resource_from_template(
    template_name="standard_96_well_plate",
    resource_name="Assay Plate #001",
    overrides={
        "attributes": {"experiment_id": "EXP001", "assay_type": "ELISA"}
    }
)

assay_plate_2 = client.create_resource_from_template(
    template_name="standard_96_well_plate",
    resource_name="Assay Plate #002",
    overrides={
        "attributes": {"experiment_id": "EXP002", "assay_type": "cell_culture"}
    }
)

# Both plates inherit standard configuration but with custom attributes
```

### Template Management Operations

**Listing and Discovery:**
```python
# List all available templates
all_templates = client.list_templates()

# Filter templates by category
plate_templates = client.list_templates(base_type="container", tags=["plate"])
reagent_templates = client.list_templates(base_type="consumable", tags=["reagent"])

# Get templates organized by category
templates_by_category = client.get_templates_by_category()
# Returns: {"container": ["plate_template", "rack_template"], "consumable": ["buffer_template"]}

# Filter by creator
lab_templates = client.list_templates(created_by="lab_manager")
```

**Template Metadata:**
```python
# Get detailed template information
template_info = client.get_template_info("standard_96_well_plate")

# Returns metadata dictionary:
# {
#   "description": "Standard 96-well polystyrene plate for assays",
#   "required_overrides": ["resource_name"],
#   "tags": ["plate", "96-well", "assay", "standard"],
#   "created_by": "lab_manager",
#   "version": "1.0.0",
#   "created_at": "2024-01-15T10:30:00Z",
#   "resource": <template_resource_object>
# }
```

**Template Updates:**
```python
# Update template metadata
updated_template = client.update_template(
    template_name="standard_96_well_plate",
    updates={
        "description": "Updated standard 96-well plate with new specifications",
        "tags": ["plate", "96-well", "assay", "standard", "v2"],
        "version": "1.1.0",
        "attributes": {"well_volume": 250}  # Updated well volume
    }
)
```

**Template Deletion:**
```python
# Remove template (permanent)
success = client.delete_template("obsolete_template")
if success:
    print("Template successfully deleted")
```

### Template Use Cases

**1. Standardized Labware:**
```python
# Create templates for common labware
tip_box_template = client.create_template(
    resource=Grid(resource_name="Standard Tip Box", rows=8, columns=12, resource_class="tips"),
    template_name="standard_tip_box",
    description="200µL tip box template",
    required_overrides=["resource_name"],
    tags=["tips", "consumable", "standard"]
)

# Create multiple tip boxes from template
for i in range(5):
    client.create_resource_from_template(
        template_name="standard_tip_box",
        resource_name=f"Tip Box #{i+1:03d}",
        overrides={"attributes": {"batch_number": f"TB{i+1:03d}"}}
    )
```

**2. Reagent Standards:**
```python
# Create reagent template
buffer_template = client.create_template(
    resource=Consumable(
        resource_name="PBS Buffer",
        resource_class="buffer",
        quantity=1000.0,
        attributes={"pH": 7.4, "concentration": "1X", "units": "mL"}
    ),
    template_name="pbs_buffer_1x",
    description="1X PBS buffer, pH 7.4",
    required_overrides=["resource_name", "quantity"],
    tags=["buffer", "pbs", "cell_culture"]
)

# Create buffer instances
buffer_stock = client.create_resource_from_template(
    template_name="pbs_buffer_1x",
    resource_name="PBS Stock #001",
    overrides={"quantity": 5000.0, "attributes": {"lot_number": "PBS2024001"}}
)
```

**3. Container Hierarchies:**
```python
# Template for plate storage systems
storage_template = client.create_template(
    resource=Stack(resource_name="Plate Storage", resource_class="storage", capacity=20),
    template_name="plate_storage_stack",
    description="Standard plate storage stack (20 plates)",
    required_overrides=["resource_name"],
    tags=["storage", "plate", "stack"]
)

# Create storage locations
incubator_storage = client.create_resource_from_template(
    template_name="plate_storage_stack",
    resource_name="Incubator Plate Stack",
    overrides={"attributes": {"temperature": 37, "humidity": 95}}
)
```

### Template Best Practices

**1. Use Meaningful Names and Tags:**
```python
# ✅ Good - Descriptive and searchable
client.create_template(
    resource=plate,
    template_name="corning_96_well_flat_bottom",
    tags=["plate", "96-well", "flat-bottom", "corning", "cell-culture"]
)

# ❌ Avoid - Generic and hard to find
client.create_template(resource=plate, template_name="plate1", tags=["lab"])
```

**2. Define Required Overrides:**
```python
# ✅ Good - Enforce customization of unique fields
client.create_template(
    resource=sample,
    template_name="dna_sample_template",
    required_overrides=["resource_name", "attributes.sample_id", "attributes.source"]
)

# ❌ Avoid - No required overrides may lead to duplicate names
client.create_template(resource=sample, template_name="sample_template")
```

**3. Version Your Templates:**
```python
# Version templates for tracking changes
client.create_template(
    resource=updated_plate,
    template_name="assay_plate_v2",
    description="Updated assay plate with improved specifications",
    version="2.0.0",
    tags=["plate", "assay", "v2"]
)
```

**4. Organize with Categories:**
```python
# Use consistent tag hierarchies
consumable_tags = ["consumable", "reagent", "buffer"]
labware_tags = ["labware", "plate", "96-well"]
equipment_tags = ["equipment", "analyzer", "hplc"]
```


## Resource Locking and Concurrency Control

MADSci provides comprehensive resource locking to prevent conflicts when multiple processes or nodes access the same resources concurrently.

### Basic Resource Locking

```python
from madsci.client.resource_client import ResourceClient

client = ResourceClient("http://localhost:8003")

# Acquire lock on a single resource
success = client.acquire_lock(
    resource=sample_plate,
    lock_duration=300.0,  # 5 minutes
)

if success:
    try:
        # Perform operations on the locked resource
        client.set_child(resource=sample_plate, key=(0, 0), child=new_sample)
        client.update_resource(sample_plate)
    finally:
        # Always release the lock
        client.release_lock(resource=sample_plate)
```

### Context Manager for Automatic Lock Management

The recommended approach uses context managers for automatic lock acquisition and release:

```python
# Single resource locking
with client.lock(sample_plate) as locked_plate:
    # Resource is automatically locked
    locked_plate.set_child(key=(0, 0), child=new_sample)
    locked_plate.update_resource()
    # Lock automatically released when exiting context

# Multiple resource locking (atomic)
with client.lock(reagent_bottle, sample_rack, plate_stack) as (reagent, rack, stack):
    # All resources locked atomically or operation fails
    reagent.decrease_quantity(amount=50.0)
    new_sample = rack.get_child(key=5)
    stack.push(child=finished_plate)
    # All locks released automatically
```

### Advanced Locking Patterns

**Lock Duration and Auto-Refresh:**
```python
# Custom lock duration with auto-refresh
with client.lock(
    resource=long_running_plate,
    lock_duration=60.0,     # 1 minute initial lock
    auto_refresh=True,      # Automatically extend lock if needed
) as locked_plate:
    # Perform long-running operations
    # Lock automatically refreshed every 30 seconds
    for i in range(96):  # Process each well
        process_well(locked_plate, well_position=i)
```

**Lock Status Checking:**
```python
# Check if resource is currently locked
is_locked = client.is_locked(resource=sample_plate)

if not is_locked:
    with client.lock(sample_plate) as locked_plate:
        perform_analysis(locked_plate)
else:
    print("Resource currently in use by another process")
```

### Error Handling and Lock Recovery

```python
# Manual lock management with error handling
try:
    if client.acquire_lock(resource=critical_resource, lock_duration=120.0):
        try:
            # Critical operations
            perform_critical_work(critical_resource)
        finally:
            client.release_lock(resource=critical_resource)
    else:
        raise Exception("Failed to acquire lock on critical resource")
except Exception as e:
    print(f"Operation failed: {e}")
```

### Best Practices for Resource Locking

**1. Always Use Context Managers:**
```python
# ✅ Good - Automatic cleanup
with client.lock(resource) as locked_resource:
    work_with_resource(locked_resource)

# ❌ Avoid - Manual management prone to errors
client.acquire_lock(resource)
work_with_resource(resource)
client.release_lock(resource)  # May not execute if exception occurs
```

**2. Lock Multiple Resources Atomically:**
```python
# ✅ Good - All locks acquired or none
with client.lock(plate, reagent, tip_rack) as (p, r, t):
    transfer_samples(from_plate=p, reagent=r, tips=t)

# ❌ Avoid - Deadlock potential
with client.lock(plate) as p:
    with client.lock(reagent) as r:  # Could deadlock if another process locks in reverse order
        transfer_samples(p, r)
```

**3. Use Appropriate Lock Durations:**
```python
# Short operations - brief locks
with client.lock(sample, lock_duration=30.0) as s:
    result = quick_measurement(s)

# Long operations - longer locks with auto-refresh
with client.lock(plate_stack, lock_duration=300.0, auto_refresh=True) as stack:
    process_entire_stack(stack)  # May take several minutes
```

### Integration with Node Actions

Resource locking integrates seamlessly with MADSci node actions:

```python
from madsci.node_module.node_module import RestNode
from madsci.common.types.action_types import ActionResult, ActionSucceeded

class AnalyzerNode(RestNode):

    @action
    def analyze_sample(
        self,
        sample_plate_id: str,
        sample_position: tuple[int, int]
    ) -> ActionResult:
        # Acquire lock before manipulating resources
        with self.resource_client.lock(sample_plate_id) as plate:
            # Get sample from locked plate
            sample = plate.get_child(key=sample_position)

            # Perform analysis
            result = self.instrument.analyze(sample)

            # Update sample with results
            sample.attributes["analysis_result"] = result
            plate.set_child(key=sample_position, child=sample)

            return ActionSucceeded(data=result)
```

## Resource Hierarchy Queries

The Resource Manager provides functionality to query the hierarchical relationships between resources, making it easy to understand parent-child relationships and navigate resource trees.

### Understanding Resource Hierarchy

Resources can form hierarchical structures where:
- **Parent resources** contain child resources (e.g., a plate contains samples)
- **Child resources** belong to parent resources and have a specific key/position
- **Ancestor resources** are all parents up the hierarchy chain
- **Descendant resources** are all children down the hierarchy chain

### Querying Resource Hierarchy

```python
from madsci.client.resource_client import ResourceClient

client = ResourceClient("http://localhost:8003")

# Create a hierarchy: Rack -> Plate -> Sample
rack = Grid(resource_name="Sample Rack", rows=2, columns=3, resource_class="rack")
rack = client.add_resource(rack)

plate = Grid(resource_name="96-well Plate", rows=8, columns=12, resource_class="plate")
plate = client.add_resource(plate)
client.set_child(resource=rack, key=(0, 0), child=plate)

sample = Asset(resource_name="Sample A1", resource_class="sample")
sample = client.add_resource(sample)
client.set_child(resource=plate, key=(0, 0), child=sample)

# Query hierarchy for the plate (middle of the hierarchy)
hierarchy = client.query_resource_hierarchy(plate.resource_id)

print(f"Resource ID: {hierarchy.resource_id}")
print(f"Ancestors (closest to furthest): {hierarchy.ancestor_ids}")
print(f"Descendants by parent: {hierarchy.descendant_ids}")

# Example output:
# Resource ID: 01HQ2K3M4N5P6Q7R8S9T0V1W2X
# Ancestors: ['01HQ2K3M4N5P6Q7R8S9T0V1W2Y']  # [rack_id]
# Descendants: {
#     '01HQ2K3M4N5P6Q7R8S9T0V1W2X': ['01HQ2K3M4N5P6Q7R8S9T0V1W2Z']  # plate -> [sample_id]
# }
```

### Hierarchy Query Results

The `query_resource_hierarchy` method returns a `ResourceHierarchy` object with:

- **`ancestor_ids`**: List of parent resource IDs, ordered from closest to furthest
  - `[parent_id, grandparent_id, great_grandparent_id, ...]`
  - Empty list if the resource has no parents

- **`resource_id`**: The ID of the queried resource

- **`descendant_ids`**: Dictionary mapping parent IDs to their direct child IDs
  - Recursively includes all descendant generations (children, grandchildren, great-grandchildren, etc.)
  - Only includes direct parent-child relationships (no "uncle" or "cousin" resources)
  - Key: parent resource ID, Value: list of direct child resource IDs
  - Empty dictionary if no descendants exist

### Use Cases

**1. Navigate Up the Hierarchy:**
```python
# Find all containers holding a specific sample
sample_hierarchy = client.query_resource_hierarchy(sample_id)
for ancestor_id in sample_hierarchy.ancestor_ids:
    ancestor = client.get_resource(ancestor_id)
    print(f"Sample is contained in: {ancestor.resource_name}")
```

**2. Navigate Down the Hierarchy:**
```python
# Find all contents of a container and their sub-contents
container_hierarchy = client.query_resource_hierarchy(container_id)
for parent_id, child_ids in container_hierarchy.descendant_ids.items():
    parent = client.get_resource(parent_id)
    print(f"{parent.resource_name} contains:")
    for child_id in child_ids:
        child = client.get_resource(child_id)
        print(f"  - {child.resource_name}")
```

**3. Verify Containment Relationships:**
```python
# Check if one resource is an ancestor of another
def is_ancestor(potential_ancestor_id, resource_id, client):
    hierarchy = client.query_resource_hierarchy(resource_id)
    return potential_ancestor_id in hierarchy.ancestor_ids

# Check if one resource is a descendant of another
def is_descendant(potential_descendant_id, resource_id, client):
    hierarchy = client.query_resource_hierarchy(resource_id)
    for child_ids in hierarchy.descendant_ids.values():
        if potential_descendant_id in child_ids:
            return True
    return False
```

**4. Build Resource Trees:**
```python
# Recursively build a complete resource tree
def build_resource_tree(resource_id, client, depth=0):
    resource = client.get_resource(resource_id)
    hierarchy = client.query_resource_hierarchy(resource_id)

    indent = "  " * depth
    print(f"{indent}{resource.resource_name} ({resource.resource_id})")

    # Process direct children
    if resource_id in hierarchy.descendant_ids:
        for child_id in hierarchy.descendant_ids[resource_id]:
            build_resource_tree(child_id, client, depth + 1)

# Start from a root resource
build_resource_tree(root_container_id, client)
```

### Performance Considerations

#### Hierarchy Query Optimization
- Hierarchy queries are optimized to fetch only direct parent-child relationships
- For deep hierarchies, consider caching results if querying frequently
- The query returns all direct ancestors and recursively traverses all descendants
- Use sparingly for very large resource trees with many nested levels

## Performance Optimization

### Database Performance

**Connection Pooling:**
```bash
# Increase connection pool size for high-throughput environments
export RESOURCE_MAX_CONNECTIONS=50
export RESOURCE_CONNECTION_TIMEOUT=30
```

**Query Optimization:**
```python
# Use specific filters to reduce query scope
samples = client.query_resource(
    resource_class="sample",
    attributes={"experiment_id": "EXP001"},  # Filter early
    multiple=True
)

# Avoid retrieving large result sets at once
batch_size = 100
offset = 0
while True:
    batch = client.query_resource(
        resource_class="sample",
        limit=batch_size,
        offset=offset,
        multiple=True
    )
    if not batch:
        break
    process_batch(batch)
    offset += batch_size
```

### Resource Management Performance

**Bulk Operations:**
```python
# Batch similar operations together
resource_updates = []
for sample_id, new_attributes in sample_updates.items():
    resource = client.get_resource(sample_id)
    resource.attributes.update(new_attributes)
    resource_updates.append(resource)

# Process batch
for resource in resource_updates:
    client.update_resource(resource)
```

**Container Hierarchy Optimization:**
```python
# Cache hierarchy results for repeated access
hierarchy_cache = {}

def get_cached_hierarchy(resource_id):
    if resource_id not in hierarchy_cache:
        hierarchy_cache[resource_id] = client.query_resource_hierarchy(resource_id)
    return hierarchy_cache[resource_id]

# Use for repeated hierarchy traversals
for sample_id in sample_list:
    hierarchy = get_cached_hierarchy(sample_id)
    process_ancestors(hierarchy.ancestor_ids)
```

### Lock Management Performance

**Minimize Lock Duration:**
```python
# ✅ Good - Short lock scope
hierarchy = client.query_resource_hierarchy(plate_id)  # Outside lock
with client.lock(plate) as locked_plate:
    # Only critical operations inside lock
    locked_plate.set_child(key=(row, col), child=sample)

# ❌ Avoid - Long lock duration
with client.lock(plate) as locked_plate:
    hierarchy = client.query_resource_hierarchy(plate_id)  # Unnecessary lock usage
    locked_plate.set_child(key=(row, col), child=sample)
```

**Concurrent Operations:**
```python
# Use separate threads for independent resource operations
import threading

def process_resource(resource_id):
    with client.lock(resource_id) as locked_resource:
        perform_analysis(locked_resource)

# Process multiple resources concurrently
threads = []
for resource_id in resource_list:
    thread = threading.Thread(target=process_resource, args=(resource_id,))
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join()
```

### Memory Management

**Large Container Handling:**
```python
# For containers with many children, avoid loading all at once
def process_large_container(container_id):
    container = client.get_resource(container_id)

    # Process children by key ranges instead of loading all
    if isinstance(container, Grid):
        for row in range(container.rows):
            for col in range(container.columns):
                child = container.get_child((row, col))
                if child:
                    process_child(child)
                    # Release reference to help GC
                    child = None
```

**Resource Cleanup:**
```python
# Clean up large query results
large_result_set = client.query_resource(resource_class="sample", multiple=True)
try:
    for resource in large_result_set:
        process_resource(resource)
finally:
    # Explicit cleanup for large datasets
    large_result_set.clear()
    del large_result_set
```

### Monitoring Performance

**Query Timing:**
```python
import time

start_time = time.time()
resources = client.query_resource(resource_class="sample", multiple=True)
query_time = time.time() - start_time

if query_time > 5.0:  # Log slow queries
    print(f"Slow query detected: {query_time:.2f}s for {len(resources)} resources")
```

**Lock Contention Monitoring:**
```python
lock_wait_start = time.time()
try:
    with client.lock(resource, lock_duration=30.0) as locked_resource:
        lock_wait_time = time.time() - lock_wait_start
        if lock_wait_time > 1.0:
            print(f"Lock contention: waited {lock_wait_time:.2f}s")

        perform_operation(locked_resource)
except TimeoutError:
    print("Failed to acquire lock - high contention detected")
```

### Performance Best Practices

1. **Use Specific Queries**: Always include filters to reduce result set size
2. **Batch Operations**: Group similar operations together
3. **Cache Hierarchy Results**: For repeated hierarchy traversals
4. **Minimize Lock Scope**: Keep locked sections as short as possible
5. **Monitor Query Performance**: Log slow operations for optimization
6. **Use Connection Pooling**: Configure appropriate pool sizes for your workload
7. **Clean Up Resources**: Explicitly clean up large datasets when done

## Error Handling and Troubleshooting

### Common Error Scenarios

**1. Resource Not Found:**
```python
from madsci.common.exceptions import ResourceNotFoundError

try:
    resource = client.get_resource("invalid_id")
except ResourceNotFoundError:
    print("Resource does not exist or has been removed")
```

**2. Lock Acquisition Failures:**
```python
# Handle lock timeouts gracefully
try:
    with client.lock(resource, lock_duration=30.0) as locked_resource:
        perform_operation(locked_resource)
except TimeoutError:
    print("Resource is currently locked by another process")
    # Implement retry logic or queue the operation
```

**3. Container Capacity Violations:**
```python
try:
    client.set_child(resource=full_container, key="A1", child=new_sample)
except ValueError as e:
    if "capacity" in str(e):
        print(f"Container is full: {e}")
        # Find alternative container or wait for space
```

**4. Quantity Management Errors:**
```python
try:
    client.decrease_quantity(resource=reagent, amount=1000.0)
except ValueError as e:
    if "insufficient quantity" in str(e):
        print(f"Not enough reagent available: {e}")
        # Check current quantity and reorder if needed
        current = client.get_resource(reagent.resource_id)
        print(f"Current quantity: {current.quantity}")
```

### Database Connection Issues

**Connection Failures:**
```python
from madsci.client.resource_client import ResourceClient

try:
    client = ResourceClient("http://localhost:8003")
    # Test connection
    client.get_definition()
except Exception as e:
    print(f"Failed to connect to Resource Manager: {e}")
    # Check if service is running: docker compose ps
    # Check logs: docker compose logs resource_manager
```

**Network Timeouts:**
```python
import httpx

client = ResourceClient(
    base_url="http://localhost:8003",
    timeout=30.0  # Increase timeout for slow operations
)
```

### Performance Troubleshooting

**Slow Hierarchy Queries:**
```python
# For deep hierarchies, query specific levels instead of full tree
hierarchy = client.query_resource_hierarchy(resource_id)
if len(hierarchy.ancestor_ids) > 10:
    # Consider caching results or limiting traversal depth
    pass
```

**Large Batch Operations:**
```python
# Process resources in smaller batches to avoid timeouts
resources = client.query_resource(resource_class="sample", multiple=True)
batch_size = 50

for i in range(0, len(resources), batch_size):
    batch = resources[i:i + batch_size]
    for resource in batch:
        # Process batch
        pass
```

### Resource State Issues

**Debugging Resource State:**
```python
# Check resource history for unexpected changes
history = client.query_history(resource_id=problem_resource.resource_id)
for entry in history[-5:]:  # Last 5 changes
    print(f"{entry.timestamp}: {entry.change_type} - {entry.details}")
```

**Recovering from Soft Deletes:**
```python
# Find and restore accidentally deleted resources
deleted_resources = client.query_resource(removed=True, multiple=True)
for resource in deleted_resources:
    if resource.resource_name == "important_sample":
        client.restore_deleted_resource(resource.resource_id)
        print(f"Restored: {resource.resource_name}")
```

### Docker and Service Issues

**Service Health Check:**
```bash
# Check if Resource Manager is running
curl http://localhost:8003/health

# Check service logs
docker compose logs resource_manager

# Restart if needed
docker compose restart resource_manager
```

**Database Connection:**
```bash
# Check PostgreSQL connection
docker compose exec postgres psql -U madsci -d madsci -c "\dt"

# Check Resource Manager tables
docker compose exec postgres psql -U madsci -d madsci -c "\d resources"
```

### Common Solutions

**1. Service Won't Start:**
- Verify PostgreSQL is running: `docker compose ps postgres`
- Check port availability: `lsof -i :8003`
- Review environment variables in docker-compose.yml

**2. Resource Operations Fail:**
- Verify resource exists: `client.get_resource(resource_id)`
- Check resource type compatibility