bitcrawl 0.0.1
Universal web crawling SDK for developers
Requires Python: >=3.7
Keywords: web scraping, web crawler, html parser, content extraction, contextual filtering, firecrawl alternative, website scraper, link crawler, data mining, llm preprocessing

# BitCrawl SDK 🕷️

**Universal Web Crawling SDK for Developers**

BitCrawl is a comprehensive, developer-friendly web crawling library designed to make web data extraction simple, efficient, and powerful. Whether you're building web scrapers, data analysis tools, AI applications, or any project that needs web data, BitCrawl provides the tools you need.

[![Python 3.7+](https://img.shields.io/badge/python-3.7+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI version](https://badge.fury.io/py/bitcrawl.svg)](https://badge.fury.io/py/bitcrawl)

## 🚀 Why Choose BitCrawl?

| Feature | BitCrawl | Other Crawlers |
|---------|----------|----------------|
| **Easy to Use** | ✅ Simple Python API | ❌ Complex setup |
| **Smart Filtering** | ✅ Contextual content filtering | ❌ Raw data only |
| **Multiple Formats** | ✅ JSON, CSV, TXT output | ❌ Limited formats |
| **Flexible Crawling** | ✅ 5 different crawling modes | ❌ Single approach |
| **Developer Friendly** | ✅ Both CLI and Python API | ❌ API only |
| **Cost Efficient** | ✅ Reduces data volume by 70-90% | ❌ Full data extraction |

## 🛠️ Key Features

- **🎯 Smart Content Filtering**: Reduce noise and extract only relevant content
- **📊 Multiple Output Formats**: JSON, CSV, TXT for different use cases
- **🔄 Flexible Crawling Modes**: Scrape, crawl, search, map, extract
- **📦 Advanced Processing**: Automatic chunking and text processing
- **⚡ CLI & Python API**: Use from command line or integrate into your code
- **🤝 Respectful Crawling**: Built-in rate limiting and robots.txt compliance

## 📦 Installation

```bash
pip install bitcrawl
```

## ⚡ Quick Start

### Python API

```python
from bitcrawl import BitCrawl

# Initialize
bc = BitCrawl()

# Scrape a single page
data = bc.scrape("https://example.com")

# Crawl multiple pages with filtering
data = bc.crawl(
    url="https://example.com", 
    context="pricing information",  # Filter for relevant content
    page_limit=10
)

# Search the web
results = bc.search("machine learning tutorials", page_limit=5)

# Get structured chunks for processing
chunks = bc.get_chunks(data, chunk_size=1000)
```

### Command Line Interface

```bash
# Scrape a single page
bitcrawl --link https://example.com --mode scrape --output json

# Crawl with filtering
bitcrawl --link https://example.com --mode crawl --context "pricing" --pagenumber 10 --output csv

# Search and extract
bitcrawl --link "python tutorials" --mode search --pagenumber 5 --output json
```

## 🔧 Crawling Modes

### 1. **Scrape** - Single Page Extraction
```python
data = bc.scrape("https://example.com")
```
Perfect for extracting data from specific pages.

### 2. **Crawl** - Multi-Page Website Crawling
```python
data = bc.crawl("https://example.com", page_limit=20)
```
Follows internal links to crawl entire websites.

### 3. **Search** - Web Search Integration
```python
results = bc.search("machine learning", page_limit=10)
```
Search the web and extract content from results.

### 4. **Map** - Website Structure Mapping
```python
structure = bc.map("https://example.com")
```
Create a map of website structure and navigation.

### 5. **Extract** - Advanced Data Extraction
```python
data = bc.extract("https://example.com")
```
Extract structured data with enhanced processing.

## 🎯 Smart Content Filtering

BitCrawl's intelligent filtering reduces data volume while preserving relevant content:

```python
# Without filtering - gets everything
full_data = bc.crawl("https://docs.python.org", page_limit=10)

# With filtering - gets only relevant content (70-90% reduction)
filtered_data = bc.crawl(
    "https://docs.python.org", 
    page_limit=10,
    context="functions classes modules"  # Smart filtering
)
```

**Benefits:**
- 📉 Reduce data volume by 70-90%
- 💰 Lower processing costs
- 🎯 Higher content relevance
- ⚡ Faster data processing

## 📊 Output Formats

### JSON (Default)
```python
data = bc.scrape("https://example.com", output_format="json")
```

### CSV for Analysis
```python
data = bc.crawl("https://example.com", output_format="csv")
```

### Plain Text
```python
data = bc.search("tutorials", output_format="txt")
```

## 🔧 Advanced Features

### Text Chunking
```python
# Split content into manageable chunks
chunks = bc.get_chunks(data, chunk_size=1000, overlap=100)

# Each chunk includes content, metadata, and positioning
for chunk in chunks:
    print(f"Chunk ID: {chunk['metadata']['chunk_id']}")
    print(f"Content: {chunk['content'][:100]}...")
```

### Token Estimation
```python
# Estimate processing costs
estimated_tokens = bc.estimate_tokens(text)
print(f"Estimated tokens: {estimated_tokens}")
```

### Advanced Processing
```python
# Get comprehensive processing with multiple formats
result = bc.crawl_advanced(
    url="https://example.com",
    context="product information",
    max_pages=15,
    chunk_size=800
)

# Access different formats
chunks = result['chunks']
stats = result['stats']
vector_format = result['vector_format']  # For vector databases
```

## 📋 Common Use Cases

### 🔍 **Data Research & Analysis**
```python
# Competitive analysis
competitor_data = bc.crawl("https://competitor.com", context="pricing features")

# Market research
market_data = bc.search("industry trends 2024", page_limit=20)
```

### 🤖 **AI & Machine Learning**
```python
# Training data collection
training_data = bc.crawl("https://docs.example.com", context="tutorials examples")

# Knowledge base building
knowledge = bc.crawl_advanced("https://wiki.example.com", chunk_size=512)
```

### 📊 **Content Aggregation**
```python
# News aggregation
news = bc.search("technology news", page_limit=50)

# Documentation aggregation
docs = bc.crawl("https://docs.framework.com", context="API reference")
```

### 💼 **Business Intelligence**
```python
# Monitor competitors
updates = bc.crawl("https://competitor.com/blog", context="product updates")

# Track industry news
industry_news = bc.search("industry analysis", page_limit=25)
```

## 🛠️ Configuration Options

```python
bc = BitCrawl(
    delay=2.0,          # Delay between requests (seconds)
    verbose=True,       # Enable detailed logging
    api_key="your-key"  # For future premium features
)
```

## 📱 CLI Reference

```bash
bitcrawl [options]

Required:
  --link URL            Target URL or search query
  --mode MODE           scrape, crawl, search, map, extract
  --output FORMAT       json, csv, txt

Optional:
  --pagenumber N        Maximum pages (default: 10)
  --context TEXT        Filter content by context
  --min-relevance N     Relevance threshold (0.0-1.0)
  --delay N             Request delay in seconds
  --verbose             Enable detailed output
```

## 🧪 Examples

### Web Scraping for Analysis
```python
from bitcrawl import BitCrawl

bc = BitCrawl()

# Scrape product information
products = bc.crawl(
    "https://store.example.com",
    context="price product specifications",
    page_limit=50
)

# Export to CSV for analysis
with open("products.csv", "w") as f:
    f.write(bc._format_output(products, "csv"))
```

### Documentation Extraction
```python
# Extract API documentation
docs = bc.crawl(
    "https://api-docs.example.com",
    context="endpoints parameters examples",
    page_limit=100
)

# Get structured chunks
chunks = bc.get_chunks(docs, chunk_size=1200)
print(f"Extracted {len(chunks)} documentation sections")
```

### Market Research
```python
# Research industry trends
trends = bc.search(
    "artificial intelligence trends 2024",
    page_limit=30
)

# Get high-relevance content only
relevant_trends = [
    item for item in trends 
    if item.get('relevance_score', 0) > 0.7
]
```

## 🎯 Framework Integration

BitCrawl works seamlessly with popular frameworks:

### Data Analysis (Pandas)
```python
import pandas as pd

data = bc.crawl("https://example.com", output_format="csv")
df = pd.read_csv(io.StringIO(data))
```

### AI/ML Frameworks
```python
# For LangChain
langchain_docs = bc.for_langchain(data)

# For LlamaIndex  
llamaindex_docs = bc.for_llamaindex(data)

# For vector databases
vector_docs = bc.to_vector_format(data)
```

## 📈 Performance & Efficiency

| Metric | Typical Results |
|--------|----------------|
| **Data Reduction** | 70-90% volume decrease |
| **Relevance Score** | 85-95% content relevance |
| **Processing Speed** | 2-5 seconds per page |
| **Memory Usage** | Optimized for large datasets |

## 🚀 Coming Soon

- **🔌 Database Integration**: Direct export to popular databases
- **📊 Analytics Dashboard**: Web interface for crawling management  
- **🔄 Scheduled Crawling**: Automated recurring crawls
- **🌐 Global CDN**: Faster crawling from multiple regions
- **📱 Mobile SDK**: Native mobile app integration

## 🤝 Contributing

We welcome contributions! See our [Contributing Guide](CONTRIBUTING.md) for details.

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

## 🆘 Support

- 📖 [Documentation](https://bitcrawl.readthedocs.io)
- 🐛 [Issue Tracker](https://github.com/yourusername/bitcrawl/issues)
- 💬 [Discussions](https://github.com/yourusername/bitcrawl/discussions)
- 📧 [Email Support](mailto:support@bitcrawl.dev)

---

**Made with ❤️ for developers who need web data** 🕷️✨