markdown-scraper 1.0.0
A professional Python client for structured web scraping with fluent query building.
Requires Python: >=3.10
Keywords: automation, llm, markdown, scraper, selectors

# 🧶 Markdown Scraper SDK

[![PyPI version](https://img.shields.io/pypi/v/markdown-scraper.svg)](https://pypi.org/project/markdown-scraper/)
[![Python versions](https://img.shields.io/pypi/pyversions/markdown-scraper.svg)](https://pypi.org/project/markdown-scraper/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://docs.markdownscraper.com)

**The Enterprise RAG-Ready Extraction Engine.**
Transform the chaotic web into high-fidelity, LLM-optimized Markdown at massive scale.

---

## 🏛️ Architecture: The Micro-Orchestrator

Traditional scrapers are centralized bottlenecks. **Markdown Scraper** refactors the scraping pipeline by moving orchestration to the client-side. This SDK acts as a **Micro-Orchestrator**, managing discovery, validation, and high-concurrency farming locally.

### Why Choose Markdown Scraper?
- **⚡ Chained DSL Protocol**: A deterministic extraction language designed for sub-millisecond server-side execution.
- **🧬 Semantic Density**: Native `:semantic('auto')` support to identify "Main Content" clusters without CSS/XPath brittle logic.
- **🛡️ Verifiable Integrity**: Cryptographic checksums ensure your production pipelines are immune to structural drift.
- **🌽 Zero-Overhead Scale**: The **Map-Farm** workflow handles thousands of URLs with linear scalability.

---

## 📦 Installation

```bash
# Using uv (Recommended)
uv add markdown-scraper

# Using pip
pip install markdown-scraper
```

---

## 🛠️ The 3-Step Lifecycle: Map ➔ Farm ➔ Vectorize

### 1. Map (Discovery)
Identify the topology of a domain using sitemaps, RSS, and link-graph heuristics.

```python
from markdown_scraper import AsyncClient

async with AsyncClient(api_key="sk-...") as client:
    # Discover high-value URLs in seconds
    urls = await client.discover("https://news.ycombinator.com", depth=1)
```

### 2. Farm (High-Scale Extraction)
Harvest data in parallel with managed concurrency and automatic retries.

```python
async for result in client.farm(urls, concurrency=25):
    if result["status"] == "success":
        print(f"✅ Extracted: {result['url']}")
```

### 3. Precision (The Q Builder)
Perform surgical extractions with cleanups and semantic pivoting.

```python
from markdown_scraper import Q

# Define a surgical extraction rule
query = (
    Q(".main-content")                # Target
    .has("article")                   # Validate
    .cleanup(".ads", ".popup")        # Mutate: Remove noise
    .near("Pricing")                  # Pivot context
    .markdown()                       # Transform
)
```

---

## 🛡️ Enterprise-Grade Production

### Schema Validation & Checksums
Protect your RAG pipeline from breaking. Validated schemas generate a unique **Checksum** that bypasses DSL parsing on the server, significantly reducing latency by utilizing a pre-compiled AST (Abstract Syntax Tree).

```python
schema = {"content": query}
validation = await client.validate_schema(schema)

# Achievement: Sub-millisecond matching & execution
result = await client.extract(url, schema=schema, checksum=validation["checksum"])
```

### LLM Framework Support
Native patterns for:
- 🤖 **OpenAI**: Feed clean markdown directly into GPT-4o.
- 🦜 **LangChain**: High-performance `DocumentLoader` implementation.
- 🦙 **LlamaIndex**: Seamless `BaseReader` integration for RAG.

---

## 📖 Documentation & Technical Guides

- **[Quickstart Guide](docs/quickstart.md)** — Get running in 60 seconds.
- **[Architecture & Scale](docs/scale.md)** — Deep dive into the Map-Farm pattern.
- **[Chained DSL Protocol](docs/dsl.md)** — Understanding the high-performance extraction language.
- **[Advanced Patterns](docs/advanced-patterns.md)** — 15+ real-world scraping strategies.
- **[API Reference](docs/api.md)** — Full technical documentation.

---

## 📄 License

Apache 2.0 © [markdownscraper.com](https://markdownscraper.com)