Module ImportFrom alias alias alias ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Expr Str Pass decorator_list FunctionDef arguments args NameParam defaults body Expr Str Pass decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore Num Assign NameStore Num Assign NameStore Num Assign NameStore Num FunctionDef arguments args NameParam NameParam defaults body Expr Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore Str FunctionDef arguments args NameParam defaults body Expr Str Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore Str decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore Str decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str Raise NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam defaults body Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Raise NameLoad decorator_list NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Raise Call NameLoad Str decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Raise NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Dict decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return Dict decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return Dict decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Pass decorator_list decorator_list
N/A abc ABCMeta abstractmethod abstractproperty StartStopMixin N/A object N/A frontier_start N/A N/A self N/A N/A N/A 
        Called when the frontier starts, see :ref:`starting/stopping the frontier <frontier-start-stop>`.
         N/A N/A frontier_stop N/A N/A self N/A N/A N/A 
        Called when the frontier stops, see :ref:`starting/stopping the frontier <frontier-start-stop>`.
         N/A N/A N/A Metadata N/A StartStopMixin N/A N/A Interface definition for a frontier metadata class. This class is responsible for storing documents metadata,
    including content and optimized for write-only data flow. N/A __metaclass__ ABCMeta add_seeds N/A N/A self seeds N/A N/A N/A 
        This method is called when new seeds are added to the frontier.

        :param list seeds: A list of :class:`Request <frontera.core.models.Request>` objects.
         N/A N/A abstractmethod page_crawled N/A N/A self response links N/A N/A N/A 
        This method is called each time a page has been crawled.

        :param object response: The :class:`Response <frontera.core.models.Response>` object for the crawled page.
        :param list links: A list of :class:`Request <frontera.core.models.Request>` objects generated from         the links extracted for the crawled page.
         N/A N/A abstractmethod request_error N/A N/A self page error N/A N/A N/A 
        This method is called each time an error occurs when crawling a page

        :param object request: The crawled with error :class:`Request <frontera.core.models.Request>` object.
        :param string error: A string identifier for the error.
         N/A N/A abstractmethod N/A Queue N/A StartStopMixin N/A N/A Interface definition for a frontier queue class. The queue has priorities and partitions. N/A __metaclass__ ABCMeta get_next_requests N/A N/A self max_n_requests partition_id N/A kwargs N/A N/A 
        Returns a list of next requests to be crawled, and excludes them from internal storage.

        :param int max_next_requests: Maximum number of requests to be returned by this method.
        :param dict kwargs: A parameters from downloader component.

        :return: list of :class:`Request <frontera.core.models.Request>` objects.
         N/A NotImplementedError N/A abstractmethod schedule N/A N/A self batch N/A N/A N/A 
        Schedules a new documents for download from batch, and updates score in metadata.

        :param batch: list of tuples(fingerprint, score, request, schedule), if ``schedule`` is True, then document
            needs to be scheduled for download, False - only update score in metadata.
         N/A NotImplementedError N/A abstractmethod count N/A N/A self N/A N/A N/A 
        Returns count of documents in the queue.

        :return: int
         N/A NotImplementedError N/A abstractmethod N/A States N/A StartStopMixin N/A N/A Interface definition for a document states management class. This class is responsible for providing actual
    documents state, and persist the state changes in batch-oriented manner. N/A __metaclass__ ABCMeta N/A NOT_CRAWLED 0 N/A QUEUED 1 N/A CRAWLED 2 N/A ERROR 3 update_cache N/A N/A self objs N/A N/A N/A 
        Reads states from meta['state'] field of request in objs and stores states in internal cache.

        :param objs: list or tuple of :class:`Request <frontera.core.models.Request>` objects.
         N/A abstractmethod set_states N/A N/A self objs N/A N/A N/A 
        Sets meta['state'] field from cache for every request in objs.

        :param objs: list or tuple of :class:`Request <frontera.core.models.Request>` objects.
         N/A NotImplementedError N/A abstractmethod flush N/A N/A self force_clear N/A N/A N/A 
        Flushes internal cache to storage.

        :param force_clear: boolean, True - signals to clear cache after flush
         N/A NotImplementedError N/A abstractmethod fetch N/A N/A self fingerprints N/A N/A N/A 
        Get states from the persistent storage to internal cache.

        :param fingerprints: list document fingerprints, which state to read
         N/A NotImplementedError N/A abstractmethod N/A Component N/A Metadata N/A N/A 
    Interface definition for a frontier component
    The :class:`Component <frontera.core.components.Component>` object is the base class for frontier
    :class:`Middleware <frontera.core.components.Middleware>` and
    :class:`Backend <frontera.core.components.Backend>` objects.

    :class:`FrontierManager <frontera.core.manager.FrontierManager>` communicates with the active components
    using the hook methods listed below.

    Implementations are different for :class:`Middleware <frontera.core.components.Middleware>` and
    :class:`Backend <frontera.core.components.Backend>` objects, therefore methods are not fully described here
    but in their corresponding section.

     N/A __metaclass__ ABCMeta N/A component_name Base Component name N/A N/A self N/A N/A N/A 
        The component name
         N/A N/A self component_name N/A property from_manager N/A N/A cls manager N/A N/A N/A 
        Class method called from :class:`FrontierManager <frontera.core.manager.FrontierManager>` passing the
        manager itself.

        Example of usage::

            def from_manager(cls, manager):
                return cls(settings=manager.settings)

         N/A N/A cls N/A classmethod N/A Middleware N/A Component N/A N/A Interface definition for a Frontier Middlewares N/A __metaclass__ ABCMeta N/A component_name Base Middleware N/A CanonicalSolver N/A Middleware N/A N/A Interface definition for a Frontera Canonical Solver N/A __metaclass__ ABCMeta N/A component_name Base CanonicalSolver N/A PropertiesMixin N/A object N/A queue N/A N/A self N/A N/A N/A 
        :return: associated :class:`Queue <frontera.core.components.Queue>` object
         N/A NotImplementedError N/A abstractproperty metadata N/A N/A self N/A N/A N/A 
        :return: associated :class:`Metadata <frontera.core.components.Metadata>` object
         N/A NotImplementedError N/A abstractproperty states N/A N/A self N/A N/A N/A 
        :return: associated :class:`States <frontera.core.components.States>` object
         N/A NotImplementedError N/A abstractproperty N/A Backend N/A PropertiesMixin Component N/A N/A Interface definition for frontier backend. N/A __metaclass__ ABCMeta finished N/A N/A self N/A N/A N/A 
        Quick check if crawling is finished. Called pretty often, please make sure calls are lightweight.

        :return: boolean
         N/A NotImplementedError N/A abstractmethod get_next_requests N/A N/A self max_n_requests N/A kwargs N/A N/A 
        Returns a list of next requests to be crawled.

        :param int max_next_requests: Maximum number of requests to be returned by this method.
        :param dict kwargs: A parameters from downloader component.

        :return: list of :class:`Request <frontera.core.models.Request>` objects.
         N/A NotImplementedError N/A abstractmethod N/A DistributedBackend N/A Backend N/A N/A Interface definition for distributed frontier backend. Implies using in strategy worker and DB worker. N/A __metaclass__ ABCMeta strategy_worker N/A N/A cls manager N/A N/A N/A NotImplementedError N/A classmethod db_worker N/A N/A cls manager N/A N/A N/A NotImplementedError N/A classmethod N/A Partitioner N/A object N/A N/A 
    Base class for a partitioner
     __init__ N/A N/A self partitions N/A N/A N/A 
        Initialize the partitioner

        Arguments:
            partitions: A list of available partitions (during startup)
         N/A N/A self partitions partitions N/A partition N/A N/A self key partitions N/A None N/A N/A 
        Takes a string key and num_partitions as argument and returns
        a partition to be used for the message

        Arguments:
            key: the key to use for partitioning
            partitions: (optional) a list of partitions.
         N/A N/A NotImplementedError partition function has to be implemented N/A N/A BaseCrawlingStrategy N/A object N/A N/A 
    Interface definition for a crawling strategy.

    Before calling these methods strategy worker is adding 'state' key to meta field in every
    :class:`Request <frontera.core.models.Request>` with state of the URL. Pleases refer for the states to HBaseBackend
    implementation.

    After exiting from all of these methods states from meta field are passed back and stored in the backend.
     N/A __metaclass__ ABCMeta from_worker N/A N/A cls settings N/A N/A N/A 
        Called on instantiation in strategy worker.

        :param settings: :class:`Settings <frontera.settings.Settings>` instance
        :return: new instance
         N/A NotImplementedError N/A classmethod add_seeds N/A N/A self seeds N/A N/A N/A 
        Called when add_seeds event is received from spider log.

        :param list seeds: A list of :class:`Request <frontera.core.models.Request>` objects.
        :return: dict with keys as fingerprints (as hex string) and values as float scores, if no scheduling is needed,
         no fingerprint should be returned
         N/A N/A N/A abstractmethod page_crawled N/A N/A self response links N/A N/A N/A 
        Called every time document was successfully crawled, and receiving page_crawled event from spider log.

        :param object response: The :class:`Response <frontera.core.models.Response>` object for the crawled page.
        :param list links: A list of :class:`Request <frontera.core.models.Request>` objects generated from         the links extracted for the crawled page.
        :return: dict with keys as fingerprints (as hex string) and values as float scores, if no scheduling is needed,
         no fingerprint should be returned
         N/A N/A N/A abstractmethod page_error N/A N/A self request error N/A N/A N/A 
        Called every time there was error during page downloading.

        :param object request: The fetched with error :class:`Request <frontera.core.models.Request>` object.
        :param str error: A string identifier for the error.
        :return: dict with one key as fingerprint (as hex string) and value as float score, if no scheduling is needed,
         empty dict should be returned
         N/A N/A N/A abstractmethod finished N/A N/A self N/A N/A N/A 
        Called by Strategy worker, after finishing processing each cycle of spider log. If this method returns true,
        then Strategy worker reports that crawling goal is achieved, stops and exits.

        :return: bool
         N/A False N/A close N/A N/A self N/A N/A N/A 
        Called when strategy worker is about to close crawling strategy.
         N/A N/A N/A