Repository: apify/crawlee-python
File: src/crawlee/basic_crawler/_basic_crawler.py (Python)

Code before:
    http_client: NotRequired[BaseHttpClient]
    concurrency_settings: NotRequired[ConcurrencySettings]
    max_request_retries: NotRequired[int]
    max_requests_per_crawl: NotRequired[int | None]
    max_session_rotations: NotRequired[int]
    configuration: NotRequired[Configuration]
    request_handler_timeout: NotRequired[timedelta]
    session_pool: NotRequired[SessionPool]
    use_session_pool: NotRequired[bool]
    retry_on_blocked: NotRequired[bool]
    proxy_configuration: NotRequired[ProxyConfiguration]
    statistics: NotRequired[Statistics[StatisticsState]]
    event_manager: NotRequired[EventManager]
    configure_logging: NotRequired[bool]
    _context_pipeline: NotRequired[ContextPipeline[TCrawlingContext]]
    _additional_context_managers: NotRequired[Sequence[AsyncContextManager]]
    _logger: NotRequired[logging.Logger]


class BasicCrawler(Generic[TCrawlingContext]):
    """A basic web crawler providing a framework for crawling websites.

    The `BasicCrawler` provides a low-level functionality for crawling websites, allowing users to define their
    own page download and data extraction logic. It is designed mostly to be subclassed by crawlers with specific
    purposes. In most cases, you will want to use a more specialized crawler, such as `HttpCrawler`,
    `BeautifulSoupCrawler`, `ParcelCrawler`, or `PlaywrightCrawler`. If you are an advanced user and want full
    control over the crawling process, you can subclass the `BasicCrawler` and implement the request-handling logic
    yourself.

    The crawling process begins with URLs provided by a `RequestProvider` instance. Each request is then
    handled by a user-defined `request_handler` function, which processes the page and extracts the data.

    The `BasicCrawler` includes several common features for crawling, such as:
        - automatic scaling based on the system resources,
        - retries for failed requests,
        - session management,
        - statistics tracking,
        - request routing via labels,
        - proxy rotation,
        - direct storage interaction helpers,
        - and more.
    """

    def __init__(
        self,
        *,
        request_provider: RequestProvider | None = None,
        request_handler: Callable[[TCrawlingContext], Awaitable[None]] | None = None,
        http_client: BaseHttpClient | None = None,
        concurrency_settings: ConcurrencySettings | None = None,
        max_request_retries: int = 3,

Review comment (suggestion):
```suggestion
    `BeautifulSoupCrawler`, `ParselCrawler`, or `PlaywrightCrawler`. If you are an advanced user and want full
```

Code after:
    http_client: NotRequired[BaseHttpClient]
    concurrency_settings: NotRequired[ConcurrencySettings]
    max_request_retries: NotRequired[int]
    max_requests_per_crawl: NotRequired[int | None]
    max_session_rotations: NotRequired[int]
    configuration: NotRequired[Configuration]
    request_handler_timeout: NotRequired[timedelta]
    session_pool: NotRequired[SessionPool]
    use_session_pool: NotRequired[bool]
    retry_on_blocked: NotRequired[bool]
    proxy_configuration: NotRequired[ProxyConfiguration]
    statistics: NotRequired[Statistics[StatisticsState]]
    event_manager: NotRequired[EventManager]
    configure_logging: NotRequired[bool]
    _context_pipeline: NotRequired[ContextPipeline[TCrawlingContext]]
    _additional_context_managers: NotRequired[Sequence[AsyncContextManager]]
    _logger: NotRequired[logging.Logger]


class BasicCrawler(Generic[TCrawlingContext]):
    """A basic web crawler providing a framework for crawling websites.

    The `BasicCrawler` provides a low-level functionality for crawling websites, allowing users to define their
    own page download and data extraction logic. It is designed mostly to be subclassed by crawlers with specific
    purposes. In most cases, you will want to use a more specialized crawler, such as `HttpCrawler`,
    `BeautifulSoupCrawler`, `ParselCrawler`, or `PlaywrightCrawler`. If you are an advanced user and want full
    control over the crawling process, you can subclass the `BasicCrawler` and implement the request-handling logic
    yourself.

    The crawling process begins with URLs provided by a `RequestProvider` instance. Each request is then
    handled by a user-defined `request_handler` function, which processes the page and extracts the data.

    The `BasicCrawler` includes several common features for crawling, such as:
        - automatic scaling based on the system resources,
        - retries for failed requests,
        - session management,
        - statistics tracking,
        - request routing via labels,
        - proxy rotation,
        - direct storage interaction helpers,
        - and more.
    """

    def __init__(
        self,
        *,
        request_provider: RequestProvider | None = None,
        request_handler: Callable[[TCrawlingContext], Awaitable[None]] | None = None,
        http_client: BaseHttpClient | None = None,
        concurrency_settings: ConcurrencySettings | None = None,
        max_request_retries: int = 3,