maxwell-demon 0.2.1
Semantic workflow automation for codebases and documents - CLI-first tool for content analysis, search, and code quality.
License: MIT
Requires Python: >=3.11

# Maxwell

**Semantic workflow automation for codebases and documents** - A CLI-first tool for content analysis, search, and code quality.

Maxwell provides a plugin-like workflow system where each workflow is a self-contained analyzer, generator, or searcher. All workflows use the same CLI pattern: `maxwell [--debug] <workflow-id> [options]`.

## Why Maxwell?

**The Problem:** Modern development teams face three persistent challenges:

1. **Code quality decay** - Lint errors, type issues, and tech debt accumulate faster than you can fix them manually
2. **Knowledge fragmentation** - Important decisions, bugs, and discussions are scattered across chat logs, PDFs, and documentation
3. **Context loss** - Understanding codebases requires reading thousands of files, manually tracing dependencies, and reconstructing architectural decisions

**Maxwell's Solution:**

**🔧 Automated Quality Enforcement**
```bash
maxwell validate --fix  # Auto-fix 100+ code quality issues in seconds
```
- Runs pyright, ruff, eslint, tsc across your entire codebase
- Auto-fixes common issues (imports, formatting, type hints)
- Prevents dependency drift with `maxwell validate --fix` in CI/CD
- **Real use case**: Cleaned up 200+ type errors in a 50k-line codebase in under 2 minutes

**🧠 Semantic Code Understanding - Agentic Refactoring**
```bash
maxwell agentic-refactoring  # AI finds duplicate code you missed
```

Maxwell uses **graph-based clique detection with semantic embeddings** to find duplicate code that traditional tools miss. Here's how it found a real duplicate in Maxwell's own codebase:

**Example: Detecting Duplicate Helper Methods**

Maxwell analyzed its own code and found this duplicate across two classes:

```python
# Class 1: UndocumentedExtractor (line 90)
def _has_docstring_class(self, node: cst.ClassDef) -> bool:
    """Check if class has docstring."""
    if not node.body.body:
        return False
    first_stmt = node.body.body[0]
    if isinstance(first_stmt, cst.SimpleStatementLine):
        if first_stmt.body and isinstance(first_stmt.body[0], cst.Expr):
            expr = first_stmt.body[0]
            if isinstance(expr.value, cst.SimpleString):
                return True
    return False

# Class 2: PythonDocstringAdder (line 198)
def _has_docstring_class(self, node: cst.ClassDef) -> bool:
    """Check if class already has docstring."""  # Only difference: "already"
    # ... identical 11 lines of logic ...
```

**How Maxwell found this (6-step pipeline):**

1. **Extract specs** (610 functions) - Converted each function to a text representation: signature + docstring + body structure

2. **Generate embeddings** (2560-dim vectors) - Each function spec becomes a vector using Qwen3-Embedding-4B:
   ```
   Function 1: [0.23, -0.41, 0.18, ...] (2560 numbers)
   Function 2: [0.22, -0.40, 0.19, ...] (2560 numbers)
   Cosine similarity: 0.94 (>0.75 threshold)
   ```

3. **Build similarity graph** (1122 edges) - Created edges between functions with high cosine similarity. These two functions got connected because their embeddings were 94% similar.

4. **Find cliques** (268 found) - Used NetworkX's `find_cliques()` to identify groups where *every* function is similar to *every other* function. Both `_has_docstring_class` methods formed a 2-node clique.

5. **LLM reasoning** - Qwen3-Coder-30B analyzed the clique:
   ```
   "These are exact duplicates with identical logic. Only difference is
   docstring wording ('has' vs 'already has'). Extract to shared helper
   function to eliminate 11 lines of duplicate code per call site."
   ```

6. **Quality gates** - Ran pyright, ruff to ensure refactoring would be safe (no type errors, no lint violations)

**Why cliques?** A clique means "all nodes connect to all other nodes" - not just a star pattern. This catches copy-paste cascades:
- ⭐ Star: Function A is similar to B, C, D (but B, C, D aren't similar to each other) - often false positives
- 🔗 Clique: A↔B, A↔C, A↔D, B↔C, B↔D, C↔D (all mutually similar) - true duplicates

**Benefits:**
- **Cross-file detection**: Found duplicates 74 lines apart in same file (grep with function names: 0 results)
- **Semantic understanding**: Embeddings caught the similarity despite different docstrings
- **Prevents false positives**: Cliques ensure mutual similarity, not just coincidental naming
- **Multi-language**: Works on Python, JavaScript, TypeScript in the same codebase

**Real results from Maxwell's codebase:**
- Analyzed: 610 functions, 195 classes
- Found: 268 duplicate cliques
- Example above: 92% code reduction (12 lines → 1 line per call site)

**Why this matters for agents and junior developers:**

Agentic refactoring enables **autonomous code improvement** without human expertise:

- **AI agents** (like Claude Code) can run `maxwell agentic-refactoring` to discover architectural patterns humans miss, then refactor with confidence using the generated recommendations
- **Inexperienced programmers** get expert-level DRY insights - the LLM explains *why* code is duplicated and *how* to fix it, turning every refactor into a learning opportunity
- **Quality gates prevent breaking changes** - pyright, ruff, eslint, tsc, and Maxwell's custom validators (dependency tracking, logging practices) run *before* and *after* refactoring, ensuring type safety and lint compliance

The combination of semantic analysis (what to fix) + quality gates (how to fix safely) creates a **self-service refactoring system** that even junior developers can trust.

**From code quality to development memory:**

Maxwell's agentic refactoring solves *spatial* code problems (duplicate code across files). But what about *temporal* problems - remembering why you made a decision 3 months ago, or finding that bug fix discussion buried in 28GB of chat logs?

That's where Maxwell's **chat indexing** and **semantic search** come in - turning your conversations into searchable, retrievable memory.

**🔍 Universal Semantic Search**
```bash
maxwell chat-semantic-search --query "authentication bug fix"
```
- Search your Claude chat logs, PDFs, and documentation by *meaning*, not keywords
- Content-addressable storage means same document = same hash = deduplicated
- Works across formats: Markdown, PDF, Office docs, chat logs
- **Real use case**: Found that bug fix discussion from 3 months ago in 28MB of chat logs instantly

**Beyond the context window - Flashback:**

AI models have limited context windows (typically 200K tokens). But your codebase evolution spans *months* of decisions, bugs, and discussions. Maxwell's chat indexing solves this with **semantic flashback**:

- **Temporal concept evolution**: Track how ideas evolved over time (e.g., "Why did we switch from marker-pdf to markitdown?")
- **Deep research workflow**: Pull relevant context from *all* conversations and documents, not just what fits in the current window
- **BM25 + semantic hybrid search**: Combine keyword matching (find exact error messages) with semantic understanding (find conceptually similar discussions)

When Claude Code uses Maxwell's MCP server, it can query 28GB of chat history in milliseconds - effectively giving the model **infinite memory** through retrieval-augmented generation.

**📸 Instant Codebase Context**
```bash
maxwell snapshot --output SNAPSHOT.md  # Share your entire codebase in one file
```
- Generate markdown snapshot of entire codebase (filesystem tree + file contents)
- Perfect for sharing context with AI assistants or new team members
- **Real use case**: Onboarded new developer by sharing single 50-page snapshot instead of hour-long walkthrough

**Why not just use existing tools?**
- **vs linters alone**: Maxwell runs *all* of them (pyright + ruff + eslint + tsc) with auto-fix in one command
- **vs grep/ripgrep**: Maxwell understands *meaning*, not just text patterns (semantic search with embeddings)
- **vs manual refactoring**: Maxwell uses graph analysis + LLM to find patterns humans miss
- **vs reading docs**: Maxwell generates living documentation from your actual codebase

## Features

- **Workflow-based architecture** - Extensible plugin system for analysis tasks
- **Semantic search** - Find documents and conversations by meaning using embeddings
- **Multi-format extraction** - PDFs (marker-pdf), Office docs (markitdown), Claude chat logs (.jsonl)
- **Code quality** - Multi-language validation (Python, JavaScript, TypeScript), auto-fixing, dependency checking
- **Agentic refactoring** - DRY optimization using graph analysis and LLM reasoning
- **Codebase analysis** - Architecture linting, docstring enrichment, duplicate detection
- **Codebase snapshots** - Generate markdown documentation of entire codebases
- **Content-addressable storage** - Files identified by SHA256 hash, global deduplication
- **Distributed architecture** - Each directory has `.maxwell/` (like `.git/`)
- **Comprehensive logging** - Dual logging to console (stderr) and `.maxwell/logs/maxwell.log` with `--debug` flag

## Installation

```bash
pip install -e .
```

## Quick Start

### List Available Workflows

```bash
maxwell list-workflows
```

### Validate Code Quality

Run multi-language validators (Python, JavaScript, TypeScript) to check code quality:

```bash
# Run validation with auto-fix
maxwell validate --fix

# Show results in different formats
maxwell validate --format json > results.json
maxwell validate --format human
```

Built-in validators:
- **Python**: pyright, ruff, custom Maxwell validators
- **JavaScript/TypeScript**: eslint, tsc
- **Project-wide**: dependency sync, logging practices

### Agentic Refactoring

DRY optimization using graph analysis and LLM reasoning:

```bash
# Analyze duplicate code and generate refactoring plan
maxwell agentic-refactoring
```

Creates:
- `.maxwell/quality_worksheet.json` - Multi-language quality issues (pyright, ruff, eslint, tsc)
- `.maxwell/refactoring_report.md` - Duplicate code analysis with LLM recommendations

### Generate a Codebase Snapshot

Create a markdown file with filesystem tree and file contents:

```bash
maxwell snapshot --output SNAPSHOT.md
```

### Search Chat Conversations

Find discussions using semantic search:

```bash
maxwell chat-semantic-search --query "PDF parsing marker-pdf" --top-k 5
```

Or use BM25 keyword search:

```bash
maxwell chat-bm25-search --query "marker-pdf import error" --limit 10
```

### Enrich Docstrings

Generate rich docstrings for Python and JavaScript code:

```bash
maxwell docstring-enrichment
```

### Utilities

Get timestamps and session IDs for scripting:

```bash
maxwell get-time --format iso
maxwell get-session
```

### Debug Mode

Enable verbose console output for any workflow:

```bash
maxwell --debug validate
maxwell --debug agentic-refactoring
```

## Available Workflows

Run `maxwell list-workflows` for the full list. Key workflows:

### Code Quality & Refactoring
- **validate** - Multi-language code validation (Python, JS, TS) with auto-fix
- **agentic-refactoring** - DRY optimization using graph analysis and LLM reasoning
- **architecture-lint** - Validates Maxwell's plugin architecture rules
- **docstring-enrichment** - Generate rich docstrings for Python and JavaScript

### Search & Analysis
- **chat-semantic-search** - Semantic search over indexed chat logs
- **chat-bm25-search** - Keyword search over chat conversations
- **chat-upsert** - Index chat files into MongoDB with deduplication
- **temporal-concept-evolution** - Iterative concept discovery with ontology construction
- **deep-research** - Comprehensive research across chat and filesystem data

### Documentation & Utilities
- **snapshot** - Generate codebase documentation (markdown with tree + contents)
- **tag-refactor** - Format semantic HTML tags in markdown files
- **web-scrape** - Scrape web pages using crawl4ai with Playwright
- **get-time** / **get-session** - Utility workflows for timestamps and session IDs

## Configuration

Add to `pyproject.toml`:

```toml
[tool.maxwell]
include_globs = [
    "*.md", "**/*.md",
    "*.pdf", "**/*.pdf",
    "**/.claude/projects/*.jsonl"    # Include chat logs
]
exclude_globs = [
    ".maxwell/**",
    ".git/**",
    "__pycache__/**",
    "*.pyc"
]

# Validation rules (optional)
[tool.maxwell.validation]
rules = {
    EMOJI-IN-STRING = "WARN",
    DICT-GET-FALLBACK = "WARN",
    TYPING-POOR-PRACTICE = "WARN",
    DEPENDENCY-NOT-DECLARED = "WARN",
    LOGGING-POOR-PRACTICE = "WARN"
}
```

## Architecture

```
project/
├── .maxwell/                      # Like .git/, per-directory
│   ├── data/                      # Workflow-specific data (committed to git)
│   ├── logs/                      # Execution logs
│   │   ├── maxwell.log            # Full debug log (all workflows)
│   │   └── <session>.jsonl        # LLM conversation logs
│   ├── indexes/                   # SQLite indexes (gitignored, rebuildable)
│   ├── extracted/                 # Cached extractions (gitignored)
│   ├── quality_worksheet.json     # Multi-language quality issues (agentic-refactoring)
│   ├── refactoring_report.md      # Duplicate code analysis (agentic-refactoring)
│   └── .gitignore                 # Auto-created
└── your-files/

~/.maxwell/                        # Global cache
└── cache.db                       # Shared embedding cache (deduplication)
```

### Logging

Maxwell uses dual logging:
- **Console (stderr)**: INFO level by default, use `--debug` for DEBUG level
- **File (`.maxwell/logs/maxwell.log`)**: Always DEBUG level with timestamps
- **stdout**: Kept clean for actual output (JSON, reports)

```bash
# Normal: INFO to console, DEBUG to file
maxwell validate

# Verbose: DEBUG to console and file
maxwell --debug validate

# Redirect output: logs to stderr, JSON to stdout
maxwell validate --format json > output.json
```

## What Gets Indexed

Maxwell can work with:
- **Markdown files** (`.md`)
- **PDFs** (`.pdf`) - reuses existing parsed versions when available
- **Claude chat logs** (`.jsonl` in `.claude/projects/`)
- **Office documents** (`.docx`, `.pptx`, `.xlsx`)
- **Python code** (`.py`) - semantic code analysis

## Extending Maxwell

### Plugin System

Maxwell supports external plugins without modifying core code. Plugins are loaded from:
- `~/.maxwell/plugins/` (global plugins)
- `<project>/.maxwell/plugins/` (project-specific plugins)

**Two plugin types:**

1. **Python plugins**: `.py` files with `BaseWorkflow` subclasses
2. **Script plugins**: Executable scripts with `.json` metadata

Example script plugin:

```bash
#!/usr/bin/env bash
# ~/.maxwell/plugins/hello
echo "Hello from Maxwell plugin!"
```

```json
{
  "workflow_id": "hello",
  "name": "Hello Plugin",
  "description": "Simple hello world plugin",
  "category": "utility"
}
```

See [plugins/README.md](./plugins/README.md) for detailed plugin development guide.

### Core Workflow Development

To add workflows to Maxwell core:

1. Create a Python file in `src/maxwell/workflows/`
2. Define a `BaseWorkflow` subclass with `@register_workflow` decorator
3. Implement `execute()`, `get_cli_parameters()`, and schema classes
4. Import your workflow in `src/maxwell/workflows/__init__.py`

Example:

```python
from maxwell.workflows.base import BaseWorkflow
from maxwell.registry import register_workflow

@register_workflow
class MyWorkflow(BaseWorkflow):
    workflow_id = "my-workflow"
    name = "My Custom Workflow"
    description = "Does something cool"

    def execute(self, project_root: Path, context: Dict[str, Any]) -> WorkflowResult:
        # Your logic here
        pass
```

## Development

See [AGENTS.instructions.md](./AGENTS.instructions.md) for development guidelines.

### Testing

```bash
# Run formatters and linters
isort src/
black src/
ruff check --fix src/
pyright src/
```

## Documentation

- [Development Instructions](./AGENTS.instructions.md)
- [Claude Code Config](./.claude/settings.json)
- [MCP Documentation](./docs/README_MCP.md)
- [Schema Migration Guide](./docs/SCHEMA_MIGRATION.md)

## License

See LICENSE file.