sdkrouter-cleaner 0.1.2
Local HTML cleaner for LLM pipelines — aggressive & focused cleaning, SSR hydration extraction, multiple output formats
Requires Python: >=3.10
Keywords: beautifulsoup, cleaner, html, llm, markdown, scraping

# sdkrouter-cleaner

Local HTML cleaner optimized for LLM pipelines. Aggressive DOM cleaning, SSR hydration extraction, CSS class filtering, semantic chunking, and multiple output formats — all in-process, no network required.

## Installation

```bash
pip install sdkrouter-cleaner
```

## Quick Start

```python
from sdkrouter_cleaner import HTMLCleaner

cleaner = HTMLCleaner()
result = cleaner.clean(html)

print(result.output)
print(f"Reduction: {result.stats.reduction_percent}%")
print(f"Tokens: {result.stats.original_tokens} -> {result.stats.cleaned_tokens}")
```

## Features

### HTMLCleaner (Primary API)

The main entry point with full statistics and Pydantic models:

```python
from sdkrouter_cleaner import HTMLCleaner, CleanerConfig, OutputFormat

config = CleanerConfig(
    max_tokens=5000,
    output_format=OutputFormat.MARKDOWN,
    filter_classes=True,
    class_threshold=0.5,
)

cleaner = HTMLCleaner(config)
result = cleaner.clean(html)

# Statistics
print(f"Original: {result.stats.original_size} bytes, {result.stats.original_tokens} tokens")
print(f"Cleaned:  {result.stats.cleaned_size} bytes, {result.stats.cleaned_tokens} tokens")
print(f"Reduction: {result.stats.reduction_percent}%")
print(f"Compression: {result.stats.compression_ratio}x")
print(f"Scripts removed: {result.stats.scripts_removed}")
print(f"Classes filtered: {result.stats.classes_removed}/{result.stats.classes_total}")

# Hydration data (SSR sites)
if result.has_hydration:
    products = result.hydration_data.get("products", [])
else:
    cleaned = result.html  # or result.output for formatted version
```

#### CleanerConfig Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `max_tokens` | `int` | `10000` | Target maximum tokens (100–100,000) |
| `output_format` | `OutputFormat` | `HTML` | Output: `html`, `markdown`, `aom`, `xtree` |
| `remove_scripts` | `bool` | `True` | Remove `<script>` tags |
| `remove_styles` | `bool` | `True` | Remove `<style>` tags and inline styles |
| `remove_comments` | `bool` | `True` | Remove HTML comments |
| `remove_hidden` | `bool` | `True` | Remove hidden/aria-hidden elements |
| `remove_empty` | `bool` | `True` | Remove empty container elements |
| `filter_classes` | `bool` | `True` | Filter CSS classes by semantic score |
| `class_threshold` | `float` | `0.3` | Min score to keep a class (0.0–1.0) |
| `enable_chunking` | `bool` | `True` | Semantic chunking for large content |
| `chunk_max_items` | `int` | `20` | Max items per chunk |
| `try_hydration` | `bool` | `True` | Try SSR hydration extraction first |
| `preserve_selectors` | `list[str]` | `[]` | CSS selectors to always preserve |

### Convenience Functions

```python
from sdkrouter_cleaner import clean, clean_to_json

# Quick clean with options
result = clean(html, max_tokens=5000, output_format="markdown")
print(result.output)

# Get JSON if SSR data available, otherwise cleaned HTML
data = clean_to_json(html)
if isinstance(data, dict):
    print("SSR data:", data)
else:
    print("Cleaned HTML:", data)
```

### Cleaning Pipeline

Lower-level pipeline with dataclass-based config:

```python
from sdkrouter_cleaner import CleaningPipeline, PipelineConfig, clean_html, clean_for_llm

# Pipeline with config
config = PipelineConfig(
    max_tokens=5000,
    output_format="markdown",
    enable_chunking=True,
)
pipeline = CleaningPipeline(config)
result = pipeline.process(html)

print(result.output)
print(f"Reduction: {result.reduction_percent:.1f}%")

# Convenience functions
result = clean_html(html, max_tokens=5000, output_format="markdown")
output = clean_for_llm(html)  # returns dict (SSR) or str (cleaned HTML)
```

### SSR Hydration Extraction

Extract structured data from server-side rendered pages without DOM parsing:

```python
from sdkrouter_cleaner import extract_hydration, detect_framework, Framework

# Detect framework
framework = detect_framework(html)
print(framework)  # Framework.NEXTJS_APP, Framework.NUXT3, etc.

# Extract hydration data
data = extract_hydration(html)
if data.has_data:
    print(f"Framework: {data.framework.value}")
    print(f"Method: {data.extraction_method}")
    products = data.page_props.get("products", [])
```

Supported frameworks:

| Framework | Detection Method |
|-----------|-----------------|
| Next.js (Pages) | `__NEXT_DATA__` script tag |
| Next.js (App Router) | `self.__next_f.push()` streaming |
| Nuxt 2 | `window.__NUXT__` |
| Nuxt 3 | `__NUXT_DATA__` script tag |
| SvelteKit | `__sveltekit_*` window vars |
| Remix | `window.__remixContext` |
| Gatsby | `window.___gatsby` |
| Qwik | `<script type="qwik/json">` |
| Astro | `Astro.props` |

### CSS Class Filtering

Score and filter CSS classes by semantic relevance:

```python
from sdkrouter_cleaner import score_class, filter_classes, detect_css_framework

# Score a single class
result = score_class("product-card")
print(f"{result.class_name}: {result.score} ({result.category.value})")

# Filter a list of classes
classes = ["product-card", "css-abc123", "flex", "p-4", "MuiButton-root"]
kept = filter_classes(classes, threshold=0.5)
# ["product-card"]

# Detect CSS framework in HTML
framework = detect_css_framework(html)
# "tailwind", "bootstrap", "material-ui", etc.
```

Scoring categories:

| Category | Score Range | Examples |
|----------|------------|---------|
| Semantic | 0.75–0.95 | `product`, `cart`, `price`, `title` |
| Structural | 0.65–0.85 | `header`, `nav`, `main`, `sidebar` |
| Interactive | 0.55–0.7 | `active`, `selected`, `disabled` |
| Utility | 0.4–0.5 | `flex`, `p-4`, `text-lg` (Tailwind) |
| Framework | 0.4–0.6 | `MuiButton`, `chakra-*`, `ant-*` |
| Hash | 0.0 | `css-abc123`, `sc-*`, `_a1b2c3` |

### Output Formats

#### Markdown

```python
from sdkrouter_cleaner import to_markdown, MarkdownConfig
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
md = to_markdown(soup, MarkdownConfig(include_links=True, include_images=False))
```

#### AOM YAML (Accessibility Object Model)

Playwright-style aria snapshot format:

```python
from sdkrouter_cleaner import to_aom_yaml, AOMConfig

yaml_output = to_aom_yaml(soup, AOMConfig(include_roles=True, max_text_length=100))
```

Output:
```yaml
- navigation:
  - link "Home"
  - link "Products"
- main:
  - heading "Product List" [level=1]
```

#### XTree (Hierarchical Tree)

```python
from sdkrouter_cleaner import to_xtree, XTreeConfig

tree = to_xtree(soup, XTreeConfig(max_depth=10, show_text=True, use_unicode=True))
```

Output:
```
ROOT
├─ nav#main-nav
│  ├─ a.nav-link → "Home"
│  └─ a.nav-link → "Products"
└─ main
   └─ div.product-list
      ├─ div.product-card
      │  └─ h3.title → "Product 1"
      └─ div.product-card
         └─ h3.title → "Product 2"
```

### Shadow DOM Flattening

Flatten declarative Shadow DOM for LLM visibility:

```python
from sdkrouter_cleaner import flatten_shadow_dom

flat_html = flatten_shadow_dom(html, mark_boundaries=True)
```

### DOM Downsampling (D2Snap)

Adaptive downsampling based on UI feature importance:

```python
from sdkrouter_cleaner import downsample_html, estimate_tokens

tokens_before = estimate_tokens(html)
downsampled = downsample_html(html, max_tokens=5000)
tokens_after = estimate_tokens(downsampled)
```

### Semantic Chunking

Split large pages into semantic chunks:

```python
from sdkrouter_cleaner import SemanticChunker, ChunkConfig
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
chunker = SemanticChunker(ChunkConfig(max_tokens=8000, max_items=20))
result = chunker.chunk(soup)

if result.was_chunked:
    for chunk in result.chunks:
        print(f"Chunk: {chunk.item_count} items, ~{chunk.estimated_tokens} tokens")
```

### Context Extraction

Generate optimal CSS selectors and element context:

```python
from sdkrouter_cleaner import extract_context, generate_selector, find_stable_anchor

context = extract_context(element, include_siblings=True, max_siblings=2)
print(f"CSS: {context.css_selector}")
print(f"XPath: {context.xpath}")
print(f"Anchor: {context.anchor_id}")

selector = generate_selector(element)
anchor = find_stable_anchor(element, max_depth=10)
```

## API Reference

### Exports

```python
from sdkrouter_cleaner import (
    # Primary API
    HTMLCleaner, CleanerConfig, CleanerResult, CleanerStats,
    ChunkInfo, OutputFormat, clean, clean_to_json,

    # Hydration Extraction
    HydrationExtractor, HydrationData, Framework,
    extract_hydration, detect_framework,

    # Context Window
    ContextExtractor, ContextWindow, ContextConfig,
    extract_context, find_stable_anchor, generate_selector,

    # Shadow DOM
    ShadowDOMFlattener, flatten_shadow_dom,

    # Downsampling
    D2SnapDownsampler, D2SnapConfig, downsample_html, estimate_tokens,

    # Chunking
    SemanticChunker, ChunkConfig, ChunkResult,

    # Class Scoring
    ClassSemanticScorer, score_class, filter_classes,
    clean_classes, detect_css_framework,

    # Output Formats
    AOMYAMLExporter, AOMConfig, to_aom_yaml,
    MarkdownExporter, MarkdownConfig, to_markdown,
    XTreeExporter, XTreeConfig, to_xtree,

    # Pipeline
    CleaningPipeline, PipelineConfig, PipelineResult,
    clean_html, clean_for_llm,
)
```

## Requirements

- Python >= 3.10
- beautifulsoup4 >= 4.12
- lxml >= 5.3
- pydantic >= 2.10
- markdownify >= 0.14
- tiktoken >= 0.8

## License

MIT