crawler-sdk 1.3.0
A pluggable SDK for extracting author information from academic publisher websites
Requires Python: >=3.8
Keywords: academic, author, crawler, elsevier, extraction, plugin, publisher, scilit, sdk

# Crawler SDK

A pluggable SDK for extracting author information from academic publisher websites.

## Features

- **Pluggable Architecture**: Create custom extraction strategies for any publisher
- **Dual Mode Support**: Browser-based (Playwright) and API-based extraction
- **Chain of Responsibility**: Flexible email extraction with fallback mechanisms
- **Type Safe**: Full type hints support
- **Minimal Dependencies**: Core SDK has no required dependencies

## Installation

```bash
# Core SDK only (no dependencies)
pip install crawler-sdk

# With browser automation support (Playwright)
pip install crawler-sdk[browser]

# With HTTP request support (curl_cffi)
pip install crawler-sdk[http]

# All features
pip install crawler-sdk[all]
```

## Quick Start

### Creating a Custom Strategy

```python
from crawler_sdk import ParseStrategy, AuthorInfo
from crawler_sdk.extractors import ExtractorChain, ButtonClickExtractor

class MyPublisherStrategy(ParseStrategy):
    """Custom strategy for extracting authors from MyPublisher"""

    def _build_extractor_chain(self):
        chain = ExtractorChain()
        chain.add(ButtonClickExtractor())
        return chain

    async def extract_authors(self, page):
        # Use the extractor chain
        emails = await self._extract_emails_with_chain(page)

        # Build author list
        authors = []
        for i, email in enumerate(emails, 1):
            authors.append(AuthorInfo(
                author_index=i,
                name=f"Author {i}",
                emails=[email]
            ))
        return authors

    def get_selectors(self):
        return {
            'email_button': 'button.show-email',
            'email_content': '.author-email',
            'close_modal': 'button.close'
        }
```

### Using API-based Extraction

```python
from crawler_sdk.extractors import APIExtractor, APIExtractorChain
from crawler_sdk import AuthorInfo

class MyAPIExtractor(APIExtractor):
    """Extract authors via direct API calls"""

    async def extract(self, article_data):
        doi = article_data.get('doi')
        # Make API request and parse response
        # ...
        return [AuthorInfo(author_index=1, name="John Doe", emails=["john@example.com"])]

# Use the extractor
chain = APIExtractorChain()
chain.add(MyAPIExtractor())
authors = await chain.execute({'doi': '10.1234/example'})
```

### Using Validators

```python
from crawler_sdk import validate_email, validate_doi, validate_url

# Validate email addresses
assert validate_email("user@example.com") == True
assert validate_email("invalid") == False

# Validate DOIs
assert validate_doi("10.1016/j.example.2023.001") == True

# Validate URLs
assert validate_url("https://www.example.com") == True
```

### Using Decorators

```python
from crawler_sdk import retry, rate_limit, timeout

@retry(max_attempts=3, delay=1.0, backoff=2.0)
@rate_limit(min_delay=0.5, max_delay=2.0)
@timeout(seconds=30)
def fetch_article(url):
    # Your fetch logic here
    pass
```

## Data Models

### AuthorInfo

```python
from crawler_sdk import AuthorInfo

author = AuthorInfo(
    author_index=1,
    name="John Smith",
    emails=["john.smith@example.edu"],
    affiliations=["Harvard University"],
    confidence=0.95
)

# Access properties
print(author.primary_email)  # "john.smith@example.edu"
print(author.has_email())    # True
```

### ArticleInfo

```python
from crawler_sdk import ArticleInfo

article = ArticleInfo(
    title="Machine Learning in Healthcare",
    doi="10.1016/j.example.2024.001",
    url="https://example.com/article",
    authors_raw=["John Smith", "Jane Doe"],
    publisher="Elsevier",
    year=2024
)
```

## Available Extractors

### Browser-based (require Playwright)

- `ButtonClickExtractor` - Click buttons to reveal hidden emails
- `DirectTextExtractor` - Extract emails from visible page text
- `MetadataExtractor` - Extract from meta tags and JSON-LD
- `JavaScriptExtractor` - Execute JS to decode obfuscated emails

### API-based

- `APIExtractor` - Base class for custom API extractors
- `APIExtractorChain` - Chain multiple API extractors

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    Your Application                      │
├─────────────────────────────────────────────────────────┤
│                     Crawler SDK                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────┐  │
│  │ Strategies  │  │ Extractors  │  │     Types       │  │
│  │             │  │             │  │                 │  │
│  │ ParseStrategy│ │ EmailExtract│  │ AuthorInfo     │  │
│  │             │  │ ExtractChain│  │ ArticleInfo    │  │
│  └─────────────┘  │ APIExtractor│  │ CrawlResult    │  │
│                   └─────────────┘  └─────────────────┘  │
├─────────────────────────────────────────────────────────┤
│                  Optional Dependencies                   │
│  ┌─────────────┐                 ┌───────────────────┐  │
│  │ Playwright  │                 │    curl_cffi      │  │
│  │ (browser)   │                 │    (API)          │  │
│  └─────────────┘                 └───────────────────┘  │
└─────────────────────────────────────────────────────────┘
```

## License

MIT License