crawlit 0.2.0
Modular, Ethical Python Web Crawler
License: MIT
Requires Python: >=3.8

# 🕷️ crawlit - Modular, Ethical Python Web Crawler

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

A powerful, modular, and ethical web crawler built in Python. Designed for security testing, link extraction, and website structure mapping with a focus on clean architecture and extensibility.

## 🚀 Features

### Core Crawling
- **Modular Architecture**: Easily extend with custom modules and parsers
- **Synchronous & Asynchronous**: Both sync (`Crawler`) and async (`AsyncCrawler`) implementations
- **Multi-threaded Support**: Thread pool support for concurrent requests in sync crawler
- **Depth Control**: Set maximum crawl depth to prevent excessive resource usage
- **Domain Filtering**: Restrict crawling to specific domains or subdomains
- **BFS Strategy**: Breadth-first search crawling algorithm
- **Queue Management**: Pause/resume, save/load state, and queue size limits

### Ethical Crawling
- **Robots.txt Compliance**: Configurable robots.txt respect with automatic crawl-delay extraction
- **Rate Limiting**: Global and per-domain rate limiting with automatic crawl-delay support
- **Session Management**: Cookie persistence, SSL verification, and custom headers
- **Authentication**: Support for Basic/Digest Auth, OAuth tokens, API keys, and custom headers
- **Retry Logic**: Configurable retry attempts with exponential backoff

### Content Extraction
- **Advanced Table Extraction**: Extract tables with support for complex structures and cell spanning
- **Image Extraction**: Extract and analyze images including alt text and accessibility information
- **Keyword Extraction**: Identify key terms and phrases from webpage content
- **Content Extraction**: Optional metadata extraction (title, description, headings, canonical URLs)
- **Content Deduplication**: Skip pages with duplicate content using SHA-256 hashing

### Data Management
- **Page Caching**: Memory and disk-based caching with TTL support
- **Storage Management**: Optional HTML content storage (memory or disk-based)
- **Crawl Resume**: Save and resume interrupted crawls
- **Progress Tracking**: Real-time progress monitoring with callbacks
- **Multiple Output Formats**: Export results as JSON, CSV, TXT, or HTML

### Advanced Features
- **Sitemap Support**: Automatic sitemap discovery and parsing from robots.txt
- **URL Filtering**: Advanced URL filtering (patterns, extensions, query parameters)
- **Enhanced Logging**: Structured JSON logging, log rotation, configurable levels
- **Error Handling**: Comprehensive error handling with custom exceptions
- **Command Line Interface**: Simple, powerful CLI for easy usage
- **Programmatic API**: Use as a library in your own Python code

## 📋 Requirements

- Python 3.8+
- Dependencies (will be listed in requirements.txt)

## 🛠️ Installation

### From PyPI (recommended)

```bash
# Install the core library
pip install crawlit

# Install with CLI tool support
pip install crawlit[cli]
```

### From Source

```bash
# Clone the repository
git clone https://github.com/SwayamDani/crawlit.git
cd crawlit

# Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .
```

## 📘 Usage

### API Documentation

Full API documentation is available in the `docs` directory, including documentation for:
- Core crawler modules
- Extraction modules (tables, images, and keywords)
- Output formatters
- Command-line interface

To build and view the documentation:

```bash
# Install Sphinx and required packages
pip install sphinx sphinx_rtd_theme sphinxcontrib-napoleon

# Build the documentation
cd docs
make html  # On Windows: make.bat html

# View the documentation
# Open docs/_build/html/index.html in your browser
```

### As a Library in Your Python Code

#### Basic Usage

```python
from crawlit import Crawler, save_results, generate_summary_report

# Initialize the crawler with custom parameters
crawler = Crawler(
    start_url="https://example.com",
    max_depth=3,
    internal_only=True,
    user_agent="MyCustomBot/1.0",
    delay=0.5,
    respect_robots=True
)

# Start crawling
crawler.crawl()

# Get and process results
results = crawler.get_results()
print(f"Crawled {len(results)} URLs")

# Save results in different formats
save_results(results, "json", "crawl_results.json", pretty=True)
```

#### Advanced Usage with New Features

```python
from crawlit import (
    Crawler, 
    SessionManager, 
    URLFilter, 
    ProgressTracker, 
    RateLimiter,
    PageCache,
    StorageManager,
    SitemapParser,
    ContentDeduplicator,
    LoggingConfig
)

# Configure enhanced logging
logging_config = LoggingConfig(
    level="INFO",
    json_logging=True,
    log_file="crawl.log",
    rotation_size_mb=10
)
logging_config.apply()

# Set up session with authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="your-oauth-token",
    api_key="your-api-key",
    api_key_header="X-API-Key"
)

# Configure URL filtering
url_filter = URLFilter(
    allowed_patterns=[r".*example\.com.*"],
    blocked_extensions=[".pdf", ".zip"]
)

# Set up progress tracking
def progress_callback(stats):
    print(f"Progress: {stats['crawled']}/{stats['total']} URLs")

progress_tracker = ProgressTracker(callback=progress_callback)

# Configure per-domain rate limiting
rate_limiter = RateLimiter(default_delay=0.5)
rate_limiter.set_domain_delay("example.com", 1.0)  # Slower for specific domain

# Set up page caching
page_cache = PageCache(use_disk=True, cache_dir="./cache", ttl=3600)

# Configure storage (optional HTML content storage)
storage_manager = StorageManager(
    store_html_content=True,
    use_disk_storage=True,
    storage_dir="./html_storage"
)

# Set up content deduplication
content_deduplicator = ContentDeduplicator(
    min_content_length=100,
    normalize_content=True
)

# Initialize crawler with all features
crawler = Crawler(
    start_url="https://example.com",
    max_depth=3,
    session_manager=session_manager,
    url_filter=url_filter,
    progress_tracker=progress_tracker,
    rate_limiter=rate_limiter,
    page_cache=page_cache,
    storage_manager=storage_manager,
    content_deduplicator=content_deduplicator,
    use_per_domain_delay=True,
    enable_content_extraction=True,
    enable_content_deduplication=True,
    use_sitemap=True,  # Auto-discover sitemaps from robots.txt
    max_workers=4,  # Multi-threaded crawling
    max_queue_size=1000
)

# Start crawling
crawler.crawl()

# Pause and resume
crawler.pause()
# ... do something ...
crawler.resume()

# Save state for later resumption
crawler.save_state("crawl_state.json")

# Get results
results = crawler.get_results()
print(f"Crawled {len(results)} URLs")
```

#### Asynchronous Usage

```python
import asyncio
from crawlit import AsyncCrawler, AsyncRateLimiter

async def main():
    # Async crawler with async rate limiter
    rate_limiter = AsyncRateLimiter(default_delay=0.5)
    
    crawler = AsyncCrawler(
        start_url="https://example.com",
        max_depth=3,
        rate_limiter=rate_limiter,
        use_per_domain_delay=True,
        max_concurrent_requests=10
    )
    
    await crawler.crawl()
    results = crawler.get_results()
    print(f"Crawled {len(results)} URLs")

asyncio.run(main())
```

See the `examples/programmatic_usage.py` file for a complete example.

### Command Line Interface

If you installed with `pip install crawlit[cli]`, you can use the command-line interface:

```bash
# Basic usage
crawlit --url https://example.com

# Advanced options
crawlit --url https://example.com \
        --depth 3 \
        --output-format json \
        --output results.json \
        --delay 0.5 \
        --user-agent "crawlit/1.0" \
        --ignore-robots

# With new extraction features (v0.2.0+)
crawlit --url https://example.com \
        --user-agent "crawlit/2.0" \
        --extract-tables \
        --tables-output "./table_output" \
        --extract-images \
        --images-output "./image_output" \
        --extract-keywords \
        --keywords-output "keywords.json"
```

### Command Line Arguments

| Argument | Description | Default |
|----------|-------------|---------|
| `--url`, `-u` | Target website URL | Required |
| `--depth`, `-d` | Maximum crawl depth | 3 |
| `--output-format`, `-f` | Output format (json, csv, txt, html) | json |
| `--output`, `-O` | File to save results | crawl_results.json |
| `--pretty-json`, `-p` | Enable pretty-print JSON with indentation | False |
| `--ignore-robots`, `-i` | Ignore robots.txt rules | False |
| `--delay` | Delay between requests (seconds) | 0.1 |
| `--user-agent`, `-a` | Custom User-Agent string | crawlit/1.0 |
| `--allow-external`, `-e` | Allow crawling URLs outside initial domain | False |
| `--summary`, `-s` | Show a summary of crawl results | False |
| `--verbose`, `-v` | Verbose output | False |
| `--extract-keywords`, `-k` | Extract keywords from crawled pages | False |
| `--keywords-output` | File to save extracted keywords | keywords.json |
| `--max-keywords` | Maximum number of keywords to extract per page | 20 |
| `--min-word-length` | Minimum length of words to consider as keywords | 3 |
| `--extract-images`, `-img` | Extract images from crawled pages | False |
| `--images-output` | Directory to save extracted images data | image_output/ |
| `--extract-tables`, `-t` | Extract tables from crawled pages | False |
| `--tables-output` | Directory to save extracted tables | table_output/ |
| `--tables-format` | Format to save extracted tables (csv or json) | csv |
| `--min-rows` | Minimum number of rows for a table to be extracted | 1 |
| `--min-columns` | Minimum number of columns for a table to be extracted | 2 |
| `--max-table-depth` | Maximum depth to extract tables from | Same as max crawl depth |
| `--help`, `-h` | Show help message | - |

## 📊 Advanced Table Extraction

Crawlit includes powerful HTML table extraction capabilities:

```python
from crawlit.extractors.tables import extract_tables

# Extract tables with minimum rows and columns filters
tables = extract_tables(html_content, min_rows=2, min_columns=3)

# Convert tables to CSV
from crawlit.extractors.tables import tables_to_csv
tables_to_csv(tables, base_filename="extracted_tables", output_dir="output")

# Convert to dictionaries using first row as headers
from crawlit.extractors.tables import tables_to_dict
table_dicts = tables_to_dict(tables)

# Convert to JSON
from crawlit.extractors.tables import tables_to_json
tables_to_json(tables, base_filename="extracted_tables")
```

The advanced table extraction provides:
- Smart handling of `<thead>` and `<tbody>` sections
- Full support for `rowspan` and `colspan` attributes
- Consistent column count across all rows
- Thorough cell content cleaning (HTML entities, whitespace, etc.)

For examples, see `examples/enhanced_table_extraction.py` and `examples/rowspan_colspan_example.py`.

## 🔍 Keyword Extraction

Crawlit 2.0 includes sophisticated keyword extraction capabilities:

```python
from crawlit import Crawler
from crawlit.extractors.keyword_extractor import KeywordExtractor

# Option 1: Use the crawler with crawlit/2.0 user agent for automatic keyword extraction
crawler = Crawler(
    start_url="https://example.com",
    user_agent="crawlit/2.0",  # Required for keyword extraction
    max_depth=2
)
crawler.crawl()
results = crawler.get_results()

# Access extracted keywords from results
for url, data in results.items():
    if 'keywords' in data:
        print(f"Keywords for {url}: {data['keywords']}")
    if 'keyphrases' in data:
        print(f"Key phrases: {data['keyphrases']}")

# Option 2: Use the keyword extractor directly on HTML content
extractor = KeywordExtractor(min_word_length=4, max_keywords=10)
html_content = "<html><body><h1>Keyword Extraction Example</h1><p>This demonstrates advanced keyword extraction capability.</p></body></html>"

# Get keywords with scores
keywords_data = extractor.extract_keywords(html_content, include_scores=True)
print(f"Keywords: {keywords_data['keywords']}")
print(f"Scores: {keywords_data['scores']}")

# Get multi-word phrases
keyphrases = extractor.extract_keyphrases(html_content)
print(f"Key phrases: {keyphrases}")
```

The keyword extraction offers:
- Smart weighting of content based on HTML structure (headings, titles, etc.)
- Automatic filtering of common stop words
- Multi-word phrase extraction for more context-rich keywords
- Scoring based on frequency and relevance
- Integration with crawl results

For a complete example, see `examples/keyword_extraction.py`.

## 🖼️ Image Extraction

Crawlit v0.2.0 introduces comprehensive image extraction capabilities:

```python
from crawlit import Crawler
from crawlit.extractors.image_extractor import ImageTagParser

# Option 1: Using the crawler for automatic image extraction
crawler = Crawler(
    start_url="https://example.com",
    max_depth=2
)
crawler.crawl()
results = crawler.get_results()

# Access extracted images from results
for url, data in results.items():
    if 'images' in data:
        for img in data['images']:
            print(f"Image URL: {img.get('src')}")
            print(f"Alt text: {img.get('alt', 'None')}")
            if 'width' in img and 'height' in img:
                print(f"Dimensions: {img['width']}x{img['height']}")
            print(f"Decorative: {img.get('decorative', False)}")

# Option 2: Use the image extractor directly on HTML content
from html.parser import HTMLParser
parser = ImageTagParser()
html_content = "<html><body><img src='example.jpg' alt='Example'></body></html>"
parser.feed(html_content)
images = parser.images
```

The image extraction feature provides:
- Complete metadata extraction (src, alt, width, height, etc.)
- Parent element context to understand image placement
- Accessibility analysis (identifying decorative images missing alt text)
- Integration with crawl results

For a complete example, see `examples/image_extraction.py`.

## 🔐 Authentication & Session Management

Crawlit supports multiple authentication methods:

```python
from crawlit import Crawler, SessionManager

# OAuth token authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="your-oauth-token"
)

# API key authentication
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    api_key="your-api-key",
    api_key_header="X-API-Key"  # Optional, defaults to "Authorization"
)

# Basic authentication
from requests.auth import HTTPBasicAuth
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    auth=HTTPBasicAuth("username", "password")
)

# Custom headers
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    headers={"X-Custom-Header": "value"}
)

# Combine multiple methods
session_manager = SessionManager(
    user_agent="MyBot/1.0",
    oauth_token="token",
    api_key="key",
    headers={"X-Custom": "value"}
)

# Use with crawler
crawler = Crawler(
    start_url="https://example.com",
    session_manager=session_manager
)
```

## ⚡ Rate Limiting

### Per-Domain Rate Limiting

```python
from crawlit import Crawler, RateLimiter

# Create rate limiter with default delay
rate_limiter = RateLimiter(default_delay=0.5)

# Set custom delays for specific domains
rate_limiter.set_domain_delay("example.com", 1.0)
rate_limiter.set_domain_delay("api.example.com", 2.0)

# Use with crawler (automatically respects robots.txt crawl-delay)
crawler = Crawler(
    start_url="https://example.com",
    rate_limiter=rate_limiter,
    use_per_domain_delay=True
)
```

The crawler automatically extracts and respects `Crawl-delay` directives from robots.txt files.

## 🗺️ Sitemap Support

```python
from crawlit import Crawler

# Auto-discover sitemaps from robots.txt
crawler = Crawler(
    start_url="https://example.com",
    use_sitemap=True  # Automatically discovers and parses sitemaps
)

# Or provide explicit sitemap URLs
crawler = Crawler(
    start_url="https://example.com",
    use_sitemap=True,
    sitemap_urls=[
        "https://example.com/sitemap.xml",
        "https://example.com/sitemap-news.xml"
    ]
)
```

## 📊 Progress Tracking

```python
from crawlit import Crawler, ProgressTracker

def progress_callback(stats):
    print(f"Crawled: {stats['crawled']}, Failed: {stats['failed']}, Total: {stats['total']}")
    print(f"Success Rate: {stats['success_rate']:.2f}%")

progress_tracker = ProgressTracker(callback=progress_callback)

crawler = Crawler(
    start_url="https://example.com",
    progress_tracker=progress_tracker
)

crawler.crawl()

# Get final statistics
stats = progress_tracker.get_stats()
print(f"Final stats: {stats}")
```

## 🔍 URL Filtering

```python
from crawlit import Crawler, URLFilter

# Filter by patterns
url_filter = URLFilter(
    allowed_patterns=[r".*example\.com.*"],
    blocked_patterns=[r".*admin.*"]
)

# Filter by file extensions
url_filter = URLFilter(
    allowed_extensions=[".html", ".htm"],
    blocked_extensions=[".pdf", ".zip"]
)

# Use factory methods
url_filter = URLFilter.html_only()  # Only HTML pages
url_filter = URLFilter.exclude_media()  # Exclude images, videos, etc.

# Use with crawler
crawler = Crawler(
    start_url="https://example.com",
    url_filter=url_filter
)
```

## 💾 Caching & Resume

```python
from crawlit import Crawler, PageCache, CrawlResume

# Set up page caching
page_cache = PageCache(
    use_disk=True,
    cache_dir="./cache",
    ttl=3600  # Cache for 1 hour
)

crawler = Crawler(
    start_url="https://example.com",
    page_cache=page_cache
)

# Save state for resumption
crawler.save_state("crawl_state.json")

# Later, resume from saved state
crawler.load_state("crawl_state.json")
crawler.crawl()  # Continues from where it left off
```

## 🗄️ Storage Management

```python
from crawlit import Crawler, StorageManager

# Store HTML content in memory (default)
storage_manager = StorageManager(store_html_content=True)

# Disable HTML content storage (save memory)
storage_manager = StorageManager(store_html_content=False)

# Store HTML content on disk
storage_manager = StorageManager(
    store_html_content=True,
    use_disk_storage=True,
    storage_dir="./html_storage"
)

crawler = Crawler(
    start_url="https://example.com",
    storage_manager=storage_manager
)
```

## 🔄 Content Deduplication

```python
from crawlit import Crawler, ContentDeduplicator

# Set up content deduplication
content_deduplicator = ContentDeduplicator(
    min_content_length=100,  # Minimum content length to consider
    normalize_content=True   # Normalize content (remove whitespace, etc.)
)

crawler = Crawler(
    start_url="https://example.com",
    content_deduplicator=content_deduplicator,
    enable_content_deduplication=True
)

# Get deduplication statistics
stats = content_deduplicator.get_stats()
print(f"Duplicate pages found: {stats['duplicates']}")
```

## 📝 Enhanced Logging

```python
from crawlit import LoggingConfig, configure_logging

# Configure structured JSON logging
logging_config = LoggingConfig(
    level="INFO",
    json_logging=True,
    log_file="crawl.log",
    rotation_size_mb=10,  # Rotate when file reaches 10MB
    rotation_time="midnight"  # Or rotate daily at midnight
)

logging_config.apply()

# Or use convenience function
configure_logging(
    level="INFO",
    log_file="crawl.log",
    json_logging=True
)

# Use contextual logging
from crawlit import log_with_context

log_with_context("INFO", "Starting crawl", {"url": "https://example.com"})
```

## 🧵 Multi-threaded Crawling

```python
from crawlit import Crawler

# Enable multi-threaded crawling (sync crawler only)
crawler = Crawler(
    start_url="https://example.com",
    max_workers=4  # Use 4 worker threads
)

# Single-threaded (default)
crawler = Crawler(
    start_url="https://example.com",
    max_workers=1  # Or omit for default
)
```

## ⏸️ Pause & Resume

```python
from crawlit import Crawler

crawler = Crawler(start_url="https://example.com")

# Start crawling in background thread
import threading
thread = threading.Thread(target=crawler.crawl)
thread.start()

# Pause crawling
crawler.pause()

# Check if paused
if crawler.is_paused():
    print("Crawler is paused")

# Resume crawling
crawler.resume()

# Get queue statistics
stats = crawler.get_queue_stats()
print(f"Queue size: {stats['size']}")
```

## 🏗️ Project Structure

```
crawlit/
├── crawlit.py           # CLI entry point
├── requirements.txt     # Project dependencies
├── crawler/             # Core crawler modules
│   ├── __init__.py
│   ├── engine.py        # Synchronous crawler engine
│   ├── async_engine.py  # Asynchronous crawler engine
│   ├── fetcher.py       # HTTP request handling (sync)
│   ├── async_fetcher.py # HTTP request handling (async)
│   ├── parser.py        # HTML parsing and link extraction
│   └── robots.py        # Robots.txt parser with crawl-delay support
├── extractors/          # Data extraction modules
│   ├── __init__.py
│   ├── content_extractor.py  # Unified content extraction
│   ├── image_extractor.py    # Image extraction and analysis
│   ├── keyword_extractor.py  # Keyword and keyphrase extraction
│   └── tables.py            # Advanced table extraction
├── utils/               # Utility modules
│   ├── __init__.py
│   ├── session_manager.py   # Session management with auth
│   ├── url_filter.py         # Advanced URL filtering
│   ├── progress.py           # Progress tracking
│   ├── queue_manager.py      # Queue state management
│   ├── cache.py              # Page caching and resume
│   ├── storage.py            # HTML content storage
│   ├── sitemap.py            # Sitemap parsing
│   ├── rate_limiter.py       # Per-domain rate limiting
│   ├── deduplication.py      # Content deduplication
│   ├── logging_config.py     # Enhanced logging
│   └── errors.py             # Custom exceptions
├── output/              # Output formatters
│   ├── __init__.py
│   └── formatters.py    # Output formatting functions
├── examples/            # Example usage
│   ├── programmatic_usage.py  # Example of using as a library
│   ├── authenticated_crawling.py  # Authentication examples
│   └── extraction_coverage_demo.py  # Extraction features demo
└── tests/               # Unit and integration tests
    └── __init__.py
```

## 📅 Project Timeline

- **May 2025**: Initial structure and CLI setup
- **May 15, 2025**: v0.2.0 release with image extraction, table extraction, and keyword extraction features
- **June 2025**: Core functionality complete (HTTP handling, parsing, domain control)
- **June 30, 2025**: Project completion target with all core features

## 🤝 Contributing

Contributions will be welcome after the core functionality is complete. Please check back after June 30, 2025, for contribution guidelines.

## 📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 👤 Author

Built and maintained by Swayam Dani

---

**Note**: This project is under active development with completion targeted for June 30, 2025.