Module Expr Str Import alias identifier Import alias Assign NameStore ListLoad Str Str Str Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore NameLoad For NameStore Call AttributeLoad NameLoad attr NameLoad body If CompareIs NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad orelse If CompareNotEq Call NameLoad NameLoad NameLoad body Raise Call NameLoad Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad NameLoad NameLoad NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Raise Call NameLoad Str Assign NameStore BinOpMult BinOpDiv Num NameLoad Call AttributeLoad NameLoad attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assert CompareEq Call NameLoad NameLoad Call NameLoad NameLoad Assert Call AttributeLoad CompareGt NameLoad Num attr AugAssignDiv NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpMult ListLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad For NameStore Call NameLoad NameLoad body If UnaryOpNot Call AttributeLoad CompareGt NameLoad Num attr body Break Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign TupleStore SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Num If CompareIs NameLoad NameLoad body Assign NameStore AttributeLoad NameLoad attr While BoolOpAnd CompareLt NameLoad NameLoad CompareGt NameLoad Num body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword BinOpDiv NameLoad NameLoad TryExcept body Expr Yield Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad AugAssignAdd NameStore Num AugAssignAdd SubscriptStore NameLoad Index NameLoad Num handlers ExceptHandler type NameLoad body If BoolOpAnd NameLoad CompareEq SubscriptLoad NameLoad Index NameLoad Num body Assign SubscriptStore NameLoad Index SubscriptLoad NameLoad Index NameLoad Num If BoolOpAnd NameLoad UnaryOpNot NameLoad body If Call AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index SubscriptLoad NameLoad Index NameLoad Call AttributeLoad NameLoad attr NameLoad orelse Assign SubscriptStore NameLoad Index SubscriptLoad NameLoad Index NameLoad Num If Call AttributeLoad CompareGt NameLoad Num attr body AugAssignDiv SubscriptStore NameLoad Slice Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad Assign TupleStore SubscriptStore NameLoad Index NameLoad SubscriptStore NameLoad Index NameLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad Num orelse Assign SubscriptStore NameLoad Index NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad Assign NameStore Num For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad AugAssignAdd NameStore Call NameLoad NameLoad If CompareLt NameLoad NameLoad body Continue Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad If CompareIsNot NameLoad NameLoad body Expr Yield NameLoad Assign NameStore NameLoad Assign NameStore Num While NameLoad body Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad If CompareIsNot NameLoad NameLoad body Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults vararg kwarg body Expr Str For NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad body Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Num Assign NameStore ListLoad While BoolOpAnd NameLoad BoolOpOr CompareIs NameLoad NameLoad CompareLt NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num AugAssignAdd NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad Assign NameStore Call NameLoad Assign NameStore NameLoad Assign NameStore NameLoad For NameStore Call AttributeLoad SubscriptLoad NameLoad Index Num attr body Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad Assign SubscriptStore NameLoad Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Slice NameLoad If Call NameLoad SubscriptLoad NameLoad Index NameLoad body Assign NameStore NameLoad Assign SubscriptStore NameLoad Index NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Slice NameLoad If Call NameLoad SubscriptLoad NameLoad Index NameLoad body Assign NameStore NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr Num NameLoad If UnaryOpNot NameLoad body Assign NameStore NameLoad Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad body Expr Str Assert CompareEqEq Call NameLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad If CompareIsNot NameLoad NameLoad body Assign NameStore BinOpAdd Num Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad orelse Assign NameStore NameLoad If UnaryOpNot NameLoad body Assign SubscriptStore NameLoad Index NameLoad Num If Call AttributeLoad CompareGt NameLoad Num attr body AugAssignDiv SubscriptStore NameLoad Slice Call AttributeLoad NameLoad attr NameLoad Return TupleLoad Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr keyword NameLoad SubscriptLoad NameLoad Index NameLoad decorator_list
N/A N/A Utility functions for stream manipulations

.. autosummary::
    :toctree: generated/

    mux
    buffer_batch
    buffer_streamer
    batch_length

 N/A numpy np N/A six N/A __all__ N/A mux buffer_batch batch_length buffer_streamer batch_length N/A N/A batch N/A N/A N/A Determine the number of samples in a batch.

    Parameters
    ----------
    batch : dict
        A batch dictionary.  Each value must implement `len`.
        All values must have the same `len`.

    Returns
    -------
    n : int >= 0 or None
        The number of samples in this batch.
        If the batch has no fields, n is None.

    Raises
    ------
    RuntimeError
        If some two values have unequal length
     N/A n None N/A value N/A N/A six itervalues batch N/A N/A N/A n None N/A N/A n N/A len value N/A N/A N/A N/A len value n N/A N/A N/A RuntimeError Unequal field lengths N/A n N/A mux N/A N/A seed_pool n_samples k lam pool_weights with_replacement prune_empty_seeds revive N/A 256.0 None True True False N/A N/A Stochastic multiplexor for generator seeds.

    Given an array of Streamer objects, do the following:

        1. Select ``k`` seeds at random to activate
        2. Assign each activated seed a sample count ~ Poisson(lam)
        3. Yield samples from the streams by randomly multiplexing
           from the active set.
        4. When a stream is exhausted, select a new one from the pool.

    Parameters
    ----------
    seed_pool : iterable of Streamer
        The collection of Streamer objects

    n_samples : int > 0 or None
        The number of samples to generate.
        If ``None``, sample indefinitely.

    k : int > 0
        The number of streams to keep active at any time.

    lam : float > 0 or None
        Rate parameter for the Poisson distribution governing sample counts
        for individual streams.
        If ``None``, sample infinitely from each stream.

    pool_weights : np.ndarray or None
        Optional weighting for ``seed_pool``.
        If ``None``, then weights are assumed to be uniform.
        Otherwise, ``pool_weights[i]`` defines the sampling proportion
        of ``seed_pool[i]``.

        Must have the same length as ``seed_pool``.

    with_replacement : bool
        Sample Streamers with replacement.  This allows a single stream to be
        used multiple times (even simultaneously).
        If ``False``, then each Streamer is consumed at most once and never
        revisited.

    prune_empty_seeds : bool
        Disable seeds from the pool that produced no data.
        If ``True``, Streamers that previously produced no data are never
        revisited.
        Note that this may be undesireable for streams where past emptiness
        may not imply future emptiness.

    revive: bool
        If ``with_replacement`` is ``False``, setting ``revive=True``
        will re-insert previously exhausted seeds into the candidate set.

        This configuration allows a seed to be active at most once at any
        time.
     N/A n_seeds N/A len seed_pool N/A N/A n_seeds N/A N/A N/A RuntimeError Cannot mux an empty seed-pool N/A seed_distribution N/A N/A 1.0 n_seeds N/A N/A np ones n_seeds N/A N/A pool_weights None N/A N/A pool_weights N/A N/A seed_distribution copy N/A pool_weights N/A N/A np atleast_1d pool_weights N/A N/A N/A len pool_weights N/A len seed_pool N/A N/A N/A N/A pool_weights 0.0 any N/A pool_weights N/A N/A np sum pool_weights N/A streams N/A N/A None k N/A stream_weights N/A N/A np zeros k N/A stream_counts N/A N/A np zeros k dtype int N/A stream_idxs N/A N/A np zeros k dtype int N/A idx N/A range k N/A N/A N/A N/A N/A N/A seed_distribution 0 any N/A N/A N/A N/A stream_idxs N/A idx N/A N/A N/A np random choice n_seeds p seed_distribution N/A N/A N/A streams N/A idx N/A stream_weights N/A idx N/A generate_new_seed N/A stream_idxs N/A idx seed_pool pool_weights seed_distribution lam with_replacement N/A weight_norm N/A N/A np sum stream_weights N/A n 0 N/A N/A n_samples None N/A N/A n_samples N/A np inf N/A N/A N/A n n_samples N/A weight_norm 0.0 N/A N/A idx N/A N/A N/A np random choice k p N/A stream_weights weight_norm N/A N/A N/A N/A N/A N/A six advance_iterator N/A streams N/A idx N/A n 1 N/A N/A stream_counts N/A idx 1 N/A N/A N/A StopIteration N/A N/A N/A prune_empty_seeds N/A N/A stream_counts N/A idx 0 N/A N/A N/A seed_distribution N/A N/A stream_idxs N/A idx 0.0 N/A N/A revive N/A with_replacement N/A N/A N/A N/A seed_distribution any N/A N/A N/A seed_distribution N/A N/A stream_idxs N/A idx N/A N/A np max seed_distribution N/A N/A N/A seed_distribution N/A N/A stream_idxs N/A idx 1.0 N/A N/A N/A N/A seed_distribution 0 any N/A N/A N/A seed_distribution N/A N/A N/A np sum seed_distribution N/A N/A stream_idxs N/A idx N/A N/A N/A np random choice n_seeds p seed_distribution N/A N/A N/A streams N/A idx N/A stream_weights N/A idx N/A generate_new_seed N/A stream_idxs N/A idx seed_pool pool_weights seed_distribution lam with_replacement N/A N/A stream_counts N/A idx 0 N/A N/A N/A stream_weights N/A idx 0.0 N/A weight_norm N/A N/A np sum stream_weights N/A buffer_batch N/A N/A generator buffer_size N/A N/A N/A Buffer an iterable of batches into larger (or smaller) batches

    Parameters
    ----------
    generator : iterable
        The generator to buffer

    buffer_size : int > 0
        The number of examples to retain per batch.

    Yields
    ------
    batch
        A batch of size at most `buffer_size`
     N/A batches N/A N/A n 0 N/A x generator N/A N/A N/A N/A batches append x N/A n N/A batch_length x N/A N/A n buffer_size N/A N/A N/A N/A batch batches N/A __split_batches batches buffer_size N/A N/A batch None N/A N/A N/A batch N/A batch None N/A n 0 N/A batches N/A N/A N/A batch batches N/A __split_batches batches buffer_size N/A N/A batch None N/A N/A N/A batch N/A buffer_streamer N/A N/A streamer buffer_size N/A args kwargs N/A N/A Buffer a stream of batches

    Parameters
    ----------
    streamer : pescador.Streamer
        The streamer object to buffer

    buffer_size : int > 0
        the number of examples to retain per batch

    Yields
    ------
    batch
        A batch of size at most `buffer_size`

    See Also
    --------
    buffer_batch

     N/A batch N/A buffer_batch N/A N/A streamer generate args kwargs buffer_size N/A N/A N/A batch N/A __split_batches N/A N/A batches buffer_size N/A N/A N/A Split at most one batch off of a collection of batches.

    Parameters
    ----------
    batches : list
        List of batch objects

    buffer_size : int > 0 or None
        Size of the desired buffer.
        If None, the entire stream is exhausted.

    Returns
    -------
    batch, remaining_batches
        One batch of size up to buffer_size,
        and all remaining batches.

     N/A batch_size 0 N/A batch_data N/A N/A N/A batches N/A N/A buffer_size None N/A batch_size buffer_size N/A N/A N/A N/A batch_data append N/A N/A batches pop 0 N/A batch_size N/A batch_length N/A batch_data N/A -1 N/A batch N/A dict N/A residual N/A dict N/A has_residual False N/A has_data False N/A key N/A N/A N/A batch_data N/A 0 keys N/A N/A N/A batch N/A key N/A N/A np concatenate N/A N/A data N/A key N/A data batch_data N/A N/A residual N/A key N/A N/A batch N/A key N/A buffer_size N/A N/A len N/A residual N/A key N/A N/A has_residual True N/A N/A batch N/A key N/A N/A batch N/A key N/A buffer_size N/A N/A len N/A batch N/A key N/A N/A has_data True N/A has_residual N/A N/A N/A N/A batches insert 0 residual N/A N/A has_data N/A N/A batch None N/A N/A batch batches N/A generate_new_seed N/A N/A idx pool weights distribution lam with_replacement N/A 256.0 True N/A N/A Randomly select and create a stream from the pool.

    Parameters
    ----------
    pool : iterable of Streamer
        The collection of Streamer objects

    weights : np.ndarray or None
        Defines the stream sample weight of each ``pool[i]``.

        Must have the same length as ``pool``.

    distribution : np.ndarray
        Defines the probability of selecting the item '`pool[i]``.

        Notes:
        1. Must have the same length as ``pool``.
        2. ``distribution`` will be modified in-place when
        with_replacement=False.

    lam : float > 0 or None
        Rate parameter for the Poisson distribution governing sample counts
        for individual streams.
        If ``None``, sample infinitely from each stream.

    with_replacement : bool
        Sample Streamers with replacement.  This allows a single stream to be
        used multiple times (even simultaneously).
        If ``False``, then each Streamer is consumed at most once and never
        revisited.
     N/A N/A N/A len pool N/A len weights N/A len distribution N/A N/A lam None N/A N/A n_stream N/A 1 N/A N/A N/A np random poisson lam lam N/A N/A n_stream None N/A N/A with_replacement N/A N/A N/A distribution N/A idx 0.0 N/A N/A N/A N/A distribution 0 any N/A N/A N/A distribution N/A N/A N/A np sum distribution N/A N/A N/A N/A N/A pool N/A idx generate max_batches n_stream N/A weights N/A idx N/A