surfnet 0.2.0
A powerful web scraping package with integrated search, captcha handling, and hardware optimization
Requires Python: >=3.7
Keywords: web scraping, search, captcha, crawler, parallel processing, gpu acceleration, high throughput

# Surfnet

A powerful Python web scraping package with integrated search capabilities, advanced captcha handling, and hardware-optimized parallel processing.

## Features

- 🔍 **Integrated Search**: Built-in DuckDuckGo search functionality
- 🚀 **Parallel Processing**: Scrape multiple URLs simultaneously for faster results
- ⚡ **JavaScript Support**: Handle JavaScript-heavy websites with Playwright
- 🤖 **Captcha Bypassing**: Intelligent captcha detection and solving capabilities
- 🛡️ **Smart Detection**: Automatically avoids login/authentication pages
- 📊 **Content Extraction**: Target specific HTML tags for precise data collection
- 🧠 **Human-like Behavior**: Simulates natural browsing patterns to avoid detection
- 🖥️ **Hardware-Optimized**: Automatically detects system capabilities and optimizes performance
- 🔋 **GPU Acceleration**: Leverages GPU power when available for maximum speed
- 🚄 **High-Throughput**: Achieves processing rates of up to 500K pages per hour on capable hardware

## Installation

```bash
pip install surfnet
```

Or install from source:

```bash
git clone https://github.com/yourusername/surfnet.git
cd surfnet
pip install -e .
```

## Dependencies

Surfnet requires the following main dependencies:

- `duckduckgo-search`: For search functionality
- `beautifulsoup4`: For HTML parsing
- `requests`: For HTTP requests
- `playwright`: For JavaScript-heavy websites
- `concurrent-futures-extra`: For parallel processing
- `fake-useragent`: For rotating user agents
- `pydub`: For audio processing (captcha solving)
- `SpeechRecognition`: For audio captcha solving
- `psutil`: For system capability detection
- `GPUtil`: For GPU capability detection (optional)
- `ray`: For distributed processing (optional)

## Usage Examples

### Basic Usage

```python
from surfnet import Surfnet

# Initialize the scraper with automatic hardware optimization
scraper = Surfnet()  # Automatically detects optimal settings

# Search for content
results = scraper.search("Python programming", max_results=5)
for result in results:
    print(f"Title: {result['title']}")
    print(f"URL: {result['url']}")
    print(f"Snippet: {result.get('snippet', '')}")
    print("---")

# Scrape a single URL
data = scraper.scrapeurl("https://example.com", tags=["p", "h1", "article"])
print(f"Title: {data['title']}")
print(f"Content: {data['content'][:200]}...")
```

### Advanced Usage

```python
from surfnet import Surfnet

# Initialize with custom parallel processing settings
scraper = Surfnet(max_workers=8, auto_optimize=True, use_gpu=True)

# Scrape multiple URLs in parallel (automatically uses available GPUs if detected)
urls = [
    "https://example.com",
    "https://example.org",
    "https://example.net"
]
results = scraper.scrape_urls_parallel(urls, tags=["article", "p", "h1", "h2"])

# Crawl a website (max 5 pages)
site_data = scraper.crawlwebsite("https://example.com", max_pages=5)

# Handle JavaScript-heavy websites using Playwright with GPU acceleration
js_site_data = scraper.scrapeurl("https://dynamic-site.com", use_playwright=True)
```

### High-Throughput Processing

```python
from surfnet import Surfnet
from surfnet.system_profiler import SystemProfiler

# Check system capabilities and estimated throughput
profiler = SystemProfiler()
profiler.print_system_report()

# Initialize for maximum performance
scraper = Surfnet(auto_optimize=True)

# Process large batches of URLs with maximum throughput
large_url_list = ["https://example.com/page" + str(i) for i in range(1, 1001)]
results = scraper.scrape_urls_parallel(large_url_list)
```

## Hardware Optimization

Surfnet automatically detects your system's capabilities and optimizes its performance accordingly:

- **CPU**: Configures optimal number of workers based on available cores
- **Memory**: Adjusts batch sizes and processing strategies based on available RAM
- **GPU**: Leverages GPU acceleration for Playwright when available
- **Distributed Processing**: Uses Ray for advanced workload distribution on multi-core systems
- **Auto-tuning**: Dynamically adjusts parameters for maximum performance

The auto-optimization feature targets a throughput of 500K+ pages per hour on high-performance systems.

## License

MIT