semvec/pypi-packages huggingface dataset:
scrapyx-pkgs 0.2.2
ScrapyX package suite - installs scrapyx-core and scrapyx-mw by default
Requires Python: >=3.10

# scrapyx-pkgs (ScrapyX Package Suite)

This is the **main package** that provides the full ScrapyX ecosystem:

- **`scrapyx-core`** - Shared Scrapy base spider + typed service configuration registry
- **`scrapyx-mw`** - Composable Scrapy middlewares (API/session, debug, captcha, proxy rotation, etc.)

## Installation

```bash
# Install everything (recommended)
pip install scrapyx-pkgs

# Or install specific components
pip install scrapyx-pkgs[core]  # Just scrapyx-core
pip install scrapyx-pkgs[mw]     # Just scrapyx-mw
pip install scrapyx-pkgs[minimal]  # Just the namespace
```

## Why a Separate Namespace Package?

This pattern allows:
1. Independent versioning of each sub-package
2. Installing only the packages you need
3. Adding future packages under `scrapyx.*` namespace without conflicts
4. Clear separation between the umbrella (this) and actual functionality

## Structure

```
packages/scrapyx/
├── pyproject.toml          # Minimal config for namespace
├── README.md               # This file
└── src/
    └── scrapyx/
        └── __init__.py     # Empty namespace + __version__
```
scrapyx-core 0.2.2
Shared Scrapy base spider + typed service configuration registry
Requires Python: >=3.10

# scrapyx-core

Shared **base spider** and **typed service configuration** utilities for Scrapy projects.

## What you get

- `BaseServiceSpider` — production-ready base class that:
  - exposes `service_config`, `captcha_needed`, `site_key`
  - collects `items`/`errors`, has `yield_empty_item()`
  - initializes from a **typed** Service registry (from settings / optional file)

- `ServiceConfig` / `ServiceRegistry` — Pydantic models for strict config.
- `load_registry_from_settings()` — merges `settings["SERVICES"]` with optional JSON file pointed by `SCRAPYX_SERVICES_FILE`.
- `validators` — functions you can call (or wire from an extension) to **fail fast** in CI/Scrapyd.

## Minimal usage

```python
# in your spiders
from scrapyx_core.base import BaseServiceSpider

class InspectionSpider(BaseServiceSpider):
    name = "inspection"
    # use self.service_config / self.captcha_needed / self.site_key
```

Optionally, set `SCRAPYX_SERVICES_FILE=/path/to/services.json` to load/merge external config.

This package is **middleware-agnostic** and pairs well with `scrapyx-mw` (add-on + middlewares).

## Install and optional extras

Base usage (BaseServiceSpider, ServiceRegistry, config) requires only the default install:

```bash
uv add scrapyx-core
# or: pip install scrapyx-core
```

Extensions that need extra dependencies:

- **WebhookExtension** (webhook callbacks on spider close): `uv add scrapyx-core[webhook]` or `pip install scrapyx-core[webhook]`
- **CompletionPublisherExtension** (FastStream/RabbitMQ/Redis): `uv add scrapyx-core[completion]` or `pip install scrapyx-core[completion]`
- Both: `uv add scrapyx-core[webhook,completion]`