Repository: apify/crawlee-python
File: src/crawlee/_types.py (Python)

Code before:
@docs_group('Functions')
class AddRequestsFunction(Protocol):
    """Function for adding requests to the `RequestManager`, with optional filtering.

    It simplifies the process of adding requests to the `RequestManager`. It automatically opens
    the specified one and adds the provided requests.
    """

    def __call__(
        self,
        requests: Sequence[str | Request],
        **kwargs: Unpack[EnqueueLinksKwargs],
    ) -> Coroutine[None, None, None]:
        """Call dunder method.

        Args:
            requests: Requests to be added to the `RequestManager`.
            **kwargs: Additional keyword arguments.
        """


@docs_group('Functions')
class EnqueueLinksFunction(Protocol):
    """A function for enqueueing new URLs to crawl based on elements selected by a given selector and explicit requests.

    It adds explicitly passed requests to the `RequestManager`.
    It extracts URLs from the current page and enqueues them for further crawling. It allows filtering through
    selectors and other options. You can also specify labels and user data to be associated with the newly
    created `Request` objects.
    """

    def __call__(
        self,
        *,
        selector: str = 'a',
        label: str | None = None,
        user_data: dict[str, Any] | None = None,
        transform_request_function: Callable[[RequestOptions], RequestOptions | RequestTransformAction] | None = None,
        requests: Sequence[str | Request] | None = None,
        **kwargs: Unpack[EnqueueLinksKwargs],
    ) -> Coroutine[None, None, None]:
        """A call dunder method.

        Args:
            selector: A selector used to find the elements containing the links. The behaviour differs based
                on the crawler used:
                - `PlaywrightCrawler` supports CSS and XPath selectors.
                - `ParselCrawler` supports CSS selectors.
                - `BeautifulSoupCrawler` supports CSS selectors.
            label: Label for the newly created `Request` objects, used for request routing.
            user_data: User data to be provided to the newly created `Request` objects.

Review comment (bug):
I can imagine at least two instances of confusing behavior:

1. you pass in `requests` and `selector`, the function enqueues the requests (even though their elements don't match the selector), and it extracts and enqueues some additional links from the current page
2. you pass in `requests` and `transform_request_function`, but the function won't be called

I know that the docblock makes it pretty clear that this is what will happen, but I also know that there will be bug reports about this :grin: 

In my opinion, we should make two overloads and throw a runtime error if someone passes both `requests` and one or more of the other arguments.

Code after:
    """

    def __call__(
        self,
        requests: Sequence[str | Request],
        **kwargs: Unpack[EnqueueLinksKwargs],
    ) -> Coroutine[None, None, None]:
        """Call dunder method.

        Args:
            requests: Requests to be added to the `RequestManager`.
            **kwargs: Additional keyword arguments.
        """


@docs_group('Functions')
class EnqueueLinksFunction(Protocol):
    """A function for enqueueing new URLs to crawl based on elements selected by a given selector or explicit requests.

    It adds explicitly passed `requests` to the `RequestManager` or it extracts URLs from the current page and enqueues
    them for further crawling. It allows filtering through selectors and other options. You can also specify labels and
    user data to be associated with the newly created `Request` objects.

    It should not be called with `selector`, `label`, `user_data` or `transform_request_function` arguments together
    with `requests` argument.

    For even more control over the enqueued links you can use combination of `ExtractLinksFunction` and
    `AddRequestsFunction`.
    """

    @overload
    def __call__(
        self,
        *,
        selector: str | None = None,
        label: str | None = None,
        user_data: dict[str, Any] | None = None,
        transform_request_function: Callable[[RequestOptions], RequestOptions | RequestTransformAction] | None = None,
        **kwargs: Unpack[EnqueueLinksKwargs],
    ) -> Coroutine[None, None, None]: ...

    @overload
    def __call__(
        self, *, requests: Sequence[str | Request] | None = None, **kwargs: Unpack[EnqueueLinksKwargs]
    ) -> Coroutine[None, None, None]: ...

    def __call__(
        self,
        *,
        selector: str | None = None,
        label: str | None = None,