excitertools 2025.0.1
itertools with function chaining
Keywords: itertools



.. image:: https://github.com/cjrh/excitertools/workflows/Python%20application/badge.svg
    :target: https://github.com/cjrh/excitertools/actions

.. image:: https://coveralls.io/repos/github/cjrh/excitertools/badge.svg?branch=master
    :target: https://coveralls.io/github/cjrh/excitertools?branch=master

.. image:: https://img.shields.io/pypi/pyversions/excitertools.svg
    :target: https://pypi.python.org/pypi/excitertools

.. image:: https://img.shields.io/pypi/implementation/excitertools.svg
    :target: https://pypi.python.org/pypi/excitertools

.. image:: https://img.shields.io/github/tag/cjrh/excitertools.svg
    :target: https://github.com/cjrh/excitertools

.. image:: https://img.shields.io/badge/install-pip%20install%20excitertools-ff69b4.svg
    :target: https://img.shields.io/badge/install-pip%20install%20excitertools-ff69b4.svg

.. image:: https://img.shields.io/badge/dependencies-more--itertools-4488ff.svg
    :target: https://more-itertools.readthedocs.io/en/stable/

.. image:: https://img.shields.io/pypi/v/excitertools.svg
    :target: https://img.shields.io/pypi/v/excitertools.svg

.. image:: https://img.shields.io/badge/calver-YYYY.MM.MINOR-22bfda.svg
    :target: http://calver.org/

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/ambv/black


.. _more-itertools: https://more-itertools.readthedocs.io/en/stable/index.html

.. _excitertools:

excitertools
############

``itertools`` reimagined as a `fluent interface <https://en.wikipedia.org/wiki/Fluent_interface>`_.

    In software engineering, a fluent interface is an object-oriented API whose design
    relies extensively on method chaining. Its goal is to increase code legibility by
    creating a domain-specific language (DSL). The term was coined in 2005 by Eric
    Evans and Martin Fowler.

    `*Wikipedia - Fluent Interface* <https://en.wikipedia.org/wiki/Fluent_interface>`_

Note that nearly all of the ``more-itertools`` extension library is included.

Custom GPT
**********

I made a custom GPT `here at chatgpt.com <https://chatgpt.com/g/g-679767714b688191a719d7db72b72261-excitertools-helper>`_
to help with coding using excitertools as well as answering questions about it.
The GPT is set up with this complete README as its knowledge base. This is
more of an experiement than anything else, but I am curious to know whether it
is useful.

Demo
****

.. code-block:: python

    >>> range(10).map(lambda x: x*7).filter(lambda x: x % 3 == 0).collect()
    [0, 21, 42, 63]
    >>> range(10).map(lambda x: x*7).filter(lambda x: x > 0 and x % 3 == 0).collect()
    [21, 42, 63]

When the lines get long, parens can be used to split up each instruction:

.. code-block:: python

    >>> (
    ...     range(10)
    ...         .map(lambda x: x*7)
    ...         .filter(lambda x: x % 3 == 0)
    ...         .collect()
    ... )
    [0, 21, 42, 63]

What's also interesting about that is how lambda's can easily contain these
processing chains, since an entire chain is a single expression. For
example:

.. code-block:: python

    >>> names = ['caleb', 'james', 'gina']
    >>> Iter(names).map(
    ...     lambda name: (
    ...         Iter(name)
    ...             .map(lambda c: c.upper() if c in 'aeiouy' else c)
    ...             .collect(str)
    ...     )
    ... ).collect()
    ['cAlEb', 'jAmEs', 'gInA']

Something I've noticed is that ``reduce`` seems easier to use and reason
about with this fluent interface, as compared to the conventional usage
as a standalone function; also, the operator module makes ``reduce`` quite
useful for simple cases:

.. code-block:: python

    >>> from operator import add, mul
    >>> (
    ...     range(10)
    ...     .map(lambda x: x*7)
    ...     .filter(lambda x: x > 0 and x % 3 == 0)
    ...     .reduce(add)
    ... )
    126
    >>> (
    ...     range(10)
    ...     .map(lambda x: x*7)
    ...     .filter(lambda x: x > 0 and x % 3 == 0)
    ...     .reduce(mul)
    ... )
    55566

.. contents::
    :depth: 1


.. |warning| unicode:: U+26A0
.. |cool| unicode:: U+2728
.. |flux| unicode:: U+1F6E0
.. |source| unicode:: U+1F3A4
.. |sink| unicode:: U+1F3A7
.. |inf| unicode:: U+267E


How to Understand the API Documentation
#######################################

Several symbols are used to indicate things about parts of the API:

- |source| This function is a *source*, meaning that it produces data
  that will be processed in an iterator chain.
- |sink| This function is a *sink*, meaning that it consumes data that
  was processed in an iterator chain.
- |inf| This function returns an infinite iterable
- |warning| Warning - pay attention
- |flux| This API is still in flux, and might be changed or
  removed in the future
- |cool| Noteworthy; could be especially useful in many situations.

The API is arranged roughly with the module-level functions first, and
thereafter the Iter_ class itself. It is the Iter_ class that does
the work to allow these iterators to be chained together. However, the
module-level functions are more likely to be used directly and that's
why they're presented first.

The API includes wrappers for the stdlib *itertools* module, including
the "recipes" given in the *itertools* docs, as well as wrappers for
the iterators from the more-itertools_ 3rd-party package.

Module-level Replacements for Builtins
######################################

The following module-level functions, like range_, zip_ and so on, are
intended to be used as replacements for their homonymous builtins. The
only difference between these and the builtin versions is that these
return instances of the Iter_ class. Note that because Iter_ is itself
iterable, it means that the functions here can be used as drop-in
replacements.

Once you have an Iter_ instance, all of its methods become available
via function call chaining, so these toplevel functions are really only
a convenience to "get started" using the chaining syntax with minimal
upfront cost in your own code.

.. contents::
    :local:



.. _range:


|source| ``range(*args) -> "Iter[int]"``
****************************************


Replacement for the builtin ``range`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

All the same calling variations work because this function merely wraps
the original function.

.. code-block:: python

    >>> range(3).collect()
    [0, 1, 2]
    >>> range(1, 4).collect()
    [1, 2, 3]
    >>> range(1, 6, 2).collect()
    [1, 3, 5]
    >>> range(1, 101, 3).filter(lambda x: x % 7 == 0).collect()
    [7, 28, 49, 70, 91]

This example multiples, element by element, the series ``[0:5]`` with the
series ``[1:6]``. Two things to note: Firstly, Iter.zip_ is used to emit
the tuples from each series. Secondly, Iter.starmap_ is used to receive
those tuples into separate arguments in the ``lambda``.

.. code-block:: python

    >>> range(5).zip(range(1, 6)).starmap(lambda x, y: x * y).collect()
    [0, 2, 6, 12, 20]

When written in a single line as above, it can get difficult to follow
the chain of logic if there are many processing steps. Parentheses in
Python allow grouping such that expressions can be spread over multiple
lines.

This is the same example as the prior one, but formatted to be spread
over several lines. This is much clearer:

.. code-block:: python

    >>> # Written out differently
    >>> (
    ...     range(5)
    ...         .zip(range(1, 6))
    ...         .starmap(lambda x, y: x * y)
    ...         .collect()
    ... )
    [0, 2, 6, 12, 20]

If you wanted the sum instead, it isn't necessary to do the collection
at all:

.. code-block:: python

    >>> (
    ...     range(5)
    ...         .zip(range(1, 6))
    ...         .starmap(lambda x, y: x * y)
    ...         .sum()
    ... )
    40



.. _zip:


``zip(*iterables: Any) -> "Iter[Tuple[T, ...]]"``
*************************************************
Replacement for the builtin ``zip`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. _enumerate:


``enumerate(iterable) -> "Iter[Tuple[int, T]]"``
************************************************
Replacement for the builtin ``enumerate`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> import string
    >>> enumerate(string.ascii_lowercase).take(3).collect()
    [(0, 'a'), (1, 'b'), (2, 'c')]




.. _map:


``map(func: Union[Callable[..., C], str], iterable) -> "Iter[C]"``
******************************************************************
Replacement for the builtin ``map`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> result = map(lambda x: (x, ord(x)), 'caleb').dict()
    >>> assert result == {'a': 97, 'b': 98, 'c': 99, 'e': 101, 'l': 108}

    >>> result = map('x, ord(x)', 'caleb').dict()
    >>> assert result == {'a': 97, 'b': 98, 'c': 99, 'e': 101, 'l': 108}


.. _filter:


``filter(function: "Callable[[Any], bool]", iterable: "Iterable[T]") -> "Iter[T]"``
***********************************************************************************
Replacement for the builtin ``filter`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> filter(lambda x: x % 3 == 0, range(10)).collect()
    [0, 3, 6, 9]




.. _count:


|source| ``count(start=0, step: int = 1) -> "Iter[int]"``
*********************************************************


Replacement for the itertools ``count`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> count().take(5).collect()
    [0, 1, 2, 3, 4]
    >>> count(0).take(0).collect()
    []
    >>> count(10).take(0).collect()
    []
    >>> count(10).take(5).collect()
    [10, 11, 12, 13, 14]
    >>> count(1).filter(lambda x: x > 10).take(5).collect()
    [11, 12, 13, 14, 15]



.. _cycle:


``cycle(iterable) -> "Iter[T]"``
********************************
Replacement for the itertools ``count`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> cycle(range(3)).take(6).collect()
    [0, 1, 2, 0, 1, 2]
    >>> cycle([]).take(6).collect()
    []
    >>> cycle(range(3)).take(0).collect()
    []



.. _repeat:


|source| ``repeat(object: C, times=None) -> "Iter[C]"``
*******************************************************


Replacement for the itertools ``count`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> repeat('a').take(3).collect()
    ['a', 'a', 'a']
    >>> repeat([1, 2]).take(3).collect()
    [[1, 2], [1, 2], [1, 2]]
    >>> repeat([1, 2]).take(3).collapse().collect()
    [1, 2, 1, 2, 1, 2]
    >>> repeat([1, 2]).collapse().take(3).collect()
    [1, 2, 1]
    >>> repeat('a', times=3).collect()
    ['a', 'a', 'a']




This next set of functions return iterators that terminate on the shortest 
input sequence.



.. _accumulate:


``accumulate(iterable, func=None, *, initial=None)``
****************************************************
Replacement for the itertools ``accumulate`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> accumulate([1, 2, 3, 4, 5]).collect()
    [1, 3, 6, 10, 15]
    >>> if sys.version_info >= (3, 8):
    ...     output = accumulate([1, 2, 3, 4, 5], initial=100).collect()
    ...     assert output == [100, 101, 103, 106, 110, 115]
    >>> accumulate([1, 2, 3, 4, 5], operator.mul).collect()
    [1, 2, 6, 24, 120]
    >>> accumulate([]).collect()
    []
    >>> accumulate('abc').collect()
    ['a', 'ab', 'abc']
    >>> accumulate(b'abc').collect()
    [97, 195, 294]
    >>> accumulate(bytearray(b'abc')).collect()
    [97, 195, 294]



.. _chain:


``chain(*iterables: Iterable[T]) -> "Iter[T]"``
***********************************************
Replacement for the itertools ``chain`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> chain('ABC', 'DEF').collect()
    ['A', 'B', 'C', 'D', 'E', 'F']
    >>> chain().collect()
    []



.. _chain_from_iterable:


``chain_from_iterable(iterable) -> "Iter[T]"``
**********************************************
Replacement for the itertools ``chain.from_iterable`` method.
This version returns an instance of Iter_ to allow
further iterable chaining.

.. code-block:: python

    >>> chain_from_iterable(['ABC', 'DEF']).collect()
    ['A', 'B', 'C', 'D', 'E', 'F']
    >>> chain_from_iterable([]).collect()
    []



.. _compress:


``compress(data, selectors)``
*****************************
Replacement for the itertools ``compress`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> compress('ABCDEF', [1, 0, 1, 0, 1, 1]).collect()
    ['A', 'C', 'E', 'F']




.. _dropwhile:


``dropwhile(pred, iterable)``
*****************************
Replacement for the itertools ``dropwhile`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> dropwhile(lambda x: x < 4, range(6)).collect()
    [4, 5]



.. _filterfalse:


``filterfalse(pred, iterable)``
*******************************
Replacement for the itertools ``filterfalse`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> filterfalse(None, [2, 0, 3, None, 4, 0]).collect()
    [0, None, 0]



.. _groupby:


``groupby(iterable, key=None)``
*******************************
Replacement for the itertools ``groupby`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

groupby_ returns an iterator of a key and "grouper" iterable. In the
example below, we use Iter.starmap_ to collect each grouper iterable
into a list, as this makes it neater for display here in the docstring.

.. code-block:: python

    >>> (
    ...     groupby(['john', 'jill', 'anne', 'jack'], key=lambda x: x[0])
    ...         .starmap(lambda k, g: (k, list(g)))
    ...         .collect()
    ... )
    [('j', ['john', 'jill']), ('a', ['anne']), ('j', ['jack'])]




.. _islice:


``islice(iterable, *args) -> "Iter"``
*************************************
Replacement for the itertools ``islice`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> islice('ABCDEFG', 2).collect()
    ['A', 'B']
    >>> islice('ABCDEFG', 2, 4).collect()
    ['C', 'D']
    >>> islice('ABCDEFG', 2, None).collect()
    ['C', 'D', 'E', 'F', 'G']
    >>> islice('ABCDEFG', 0, None, 2).collect()
    ['A', 'C', 'E', 'G']



.. _starmap:


``starmap(func, iterable)``
***************************
Replacement for the itertools ``starmap`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> starmap(pow, [(2, 5), (3, 2), (10, 3)]).collect()
    [32, 9, 1000]



.. _takewhile:


``takewhile(pred, iterable)``
*****************************
Replacement for the itertools ``takewhile`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> takewhile(lambda x: x < 5, [1, 4, 6, 4, 1]).collect()
    [1, 4]



.. _tee:


``tee(iterable, n=2)``
**********************
Replacement for the itertools ``tee`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> a, b = tee(range(5))
    >>> a.collect()
    [0, 1, 2, 3, 4]
    >>> b.sum()
    10

It is also possible to operate on the returned iterators in the chain
but it gets quite difficult to understand:

.. code-block:: python

    >>> tee(range(5)).map(lambda it: it.sum()).collect()
    [10, 10]

In the example above we passed in range_, but with excitertools_
it's usually more natural to push data sources further left:

.. code-block:: python

    >>> range(5).tee().map(lambda it: it.sum()).collect()
    [10, 10]

Pay close attention to the above. The map_ is acting on each of the
copied iterators.



.. _zip_longest:


``zip_longest(*iterables, fillvalue=None)``
*******************************************
Replacement for the itertools ``zip_longest`` function.  This version returns
an instance of Iter_ to allow further iterable chaining.

.. code-block:: python

    >>> zip_longest('ABCD', 'xy', fillvalue='-').collect()
    [('A', 'x'), ('B', 'y'), ('C', '-'), ('D', '-')]
    >>> (
    ...     zip_longest('ABCD', 'xy', fillvalue='-')
    ...         .map(lambda tup: concat(tup, ''))
    ...         .collect()
    ... )
    ['Ax', 'By', 'C-', 'D-']
    >>> (
    ...     zip_longest('ABCD', 'xy', fillvalue='-')
    ...         .starmap(operator.add)
    ...         .collect()
    ... )
    ['Ax', 'By', 'C-', 'D-']



.. _finditer_regex:


``finditer_regex(pat: "re.Pattern[AnyStr]", s: AnyStr, flags: Union[int, re.RegexFlag] = 0) -> "Iter[AnyStr]"``
***************************************************************************************************************

Wrapper for ``re.finditer``. Returns an instance of Iter_ to allow
chaining.

.. code-block:: python

    >>> pat = r"\w+"
    >>> text = "Well hello there! How ya doin!"
    >>> finditer_regex(pat, text).map(str.lower).filter(lambda w: 'o' in w).collect()
    ['hello', 'how', 'doin']
    >>> finditer_regex(r"[A-Za-z']+", "A programmer's RegEx test.").collect()
    ['A', "programmer's", 'RegEx', 'test']
    >>> finditer_regex(r"[A-Za-z']+", "").collect()
    []
    >>> finditer_regex("", "").collect()
    ['']
    >>> finditer_regex("", "").filter(None).collect()
    []



.. _splititer_regex:


``splititer_regex(pat: "re.Pattern[AnyStr]", s: AnyStr, flags: Union[int, re.RegexFlag] = 0) -> "Iter[AnyStr]"``
****************************************************************************************************************

Lazy string splitting using regular expressions.

Most of the time you want ``str.split``. Really! That will almost
always be fastest. You might think that ``str.split`` is inefficient
because it always has to build a list, but it can do this very, very
quickly.

The lazy splitting shown here is more about supporting a particular
kind of programming model, rather than performance.

See more discussion `here <https://stackoverflow.com/questions/3862010/is-there-a-generator-version-of-string-split-in-python>`_.

.. code-block:: python

    >>> splititer_regex(r"\s", "A programmer's RegEx test.").collect()
    ['A', "programmer's", 'RegEx', 'test.']

Note that splitting at a single whitespace character will return blanks
for each found. This is different to how ``str.split()`` works.

.. code-block:: python

    >>> splititer_regex(r"\s", "aaa     bbb  \n  ccc\nddd\teee").collect()
    ['aaa', '', '', '', '', 'bbb', '', '', '', '', 'ccc', 'ddd', 'eee']

To match ``str.split()``, specify a sequence of whitespace as the
regex pattern.

.. code-block:: python

    >>> splititer_regex(r"\s+", "aaa     bbb  \n  ccc\nddd\teee").collect()
    ['aaa', 'bbb', 'ccc', 'ddd', 'eee']

Counting the whitespace

.. code-block:: python

    >>> from collections import Counter
    >>> splititer_regex(r"\s", "aaa     bbb  \n  ccc\nddd\teee").collect(Counter)
    Counter({'': 8, 'aaa': 1, 'bbb': 1, 'ccc': 1, 'ddd': 1, 'eee': 1})

Lazy splitting at newlines

.. code-block:: python

    >>> splititer_regex(r"\n", "aaa     bbb  \n  ccc\nddd\teee").collect()
    ['aaa     bbb  ', '  ccc', 'ddd\teee']

A few more examples:

.. code-block:: python

    >>> splititer_regex(r"", "aaa").collect()
    ['', 'a', 'a', 'a', '']
    >>> splititer_regex(r"", "").collect()
    ['', '']
    >>> splititer_regex(r"\s", "").collect()
    ['']
    >>> splititer_regex(r"a", "").collect()
    ['']
    >>> splititer_regex(r"\s", "aaa").collect()
    ['aaa']



.. _concat:


``concat(iterable: Iterable[AnyStr], glue: AnyStr) -> "AnyStr"``
****************************************************************
Concatenate strings, bytes and bytearrays. It is careful to avoid the
problem with single bytes becoming integers, and it looks at the value
of `glue` to know whether to handle bytes or strings.

This function can raise ``ValueError`` if called with something
other than ``bytes``, ``bytearray`` or ``str``.

.. _from_queue:


|cool| |source| ``from_queue(q: queue.Queue, timeout=None, sentinel=None) -> "Iter"``
*************************************************************************************




Wrap a queue with an iterator interface. This allows it to participate
in chaining operations. The iterator will block while waiting for
new values to appear on the queue. This is useful: it allows you
to easily and safely pass data between threads or processes, and
feed the incoming data into a pipeline.

The sentinel value, default ``None``, will terminate the iterator.

.. code-block:: python

    >>> q = queue.Queue()
    >>> # This line puts stuff onto a queue
    >>> range(10).chain([None]).map(q.put).consume()
    >>> from_queue(q).filter(lambda x: 2 < x < 9).collect()
    [3, 4, 5, 6, 7, 8]

This can be used in the same way you would normally use a queue, in
that it will block while waiting for future input. This makes it
convenient to run in a thread and wait for work. Below is a rough
sketch of how one might cobble together a thread pool using this
feature. Note the use of Iter.side_effect_ to call ``task_done()``
on the queue.

.. code-block:: python

    import queue
    from threading import Thread
    import logging
    from excitertools import from_queue

    logger = logging.getLogger(__name__)

    def process_job(job):
        result = ...
        return result

    def worker(inputs: Queue, results: Queue):
        (
            from_queue(inputs)
            .side_effect(lambda job: logger.info(f"Received job {job}")
            .map(process_job)
            .side_effect(lambda result: logger.info(f"Got result {job}")
            .into_queue(results)
            # Note that we only mark the task as done after the result
            # is added to the results queue.
            .side_effect(lambda _: inputs.task_done()
        )

    def create_job_pool(n: int) -> Tuple[Queue, Queue, Callable]:
        """Returns two queues, and a pool shutdown method. The
        shutdown function can be called to shut down the pool and
        the ``inputs`` queue. Caller is responsible for draining
        the ``results`` queue."""

        # Independent control of the sizes of the input and output
        # queues is interesting: it lets you decide how to bias
        # backpressure depending on the details of your workload.
        inputs, results = Queue(maxsize=100), Queue(maxsize=3)

        kwargs = dict(target=worker, args=(inputs, results), daemon=True)
        threads = repeat(Thread).map(lambda T: T(**kwargs)).take(n).collect()

        def shutdown():
            # Shut down each thread
            repeat(None).map(inputs.put).take(n).consume()
            inputs.join()
            Iter(threads).map(lambda t: t.join()).consume()

        return inputs, results, shutdown

Now the two queues ``inputs`` and ``results`` can be used in various
other threads to supply and consume data.



.. _fileinput:


|source| ``fileinput(files=None, inplace=False, backup="", mode="r", openhook=None, encoding=None, errors=None) -> Iter``
*************************************************************************************************************************


A wrapper around fileinput.input that returns an Excitertools Iter_ instance.

The documentation for the stdlib fileinput module is here:
    https://docs.python.org/3/library/fileinput.html

Here is an example of use:

.. code-block:: python

    >>> from excitertools import fileinput
    >>> # Read from a file
    >>> fileinput(['data.txt']).take(3).collect()                  
    ['1', '2', '3']
    >>> # Read from stdin OR files listed on the command line (sys.argv[1:])
    >>> fileinput().take(3).collect()                  
    ['1', '2', '3']

:param files: A list of filenames or '-' for stdin (default: sys.argv[1:]).
:param inplace: Whether to allow in-place editing (default: False).
:param backup: Backup extension for in-place editing (default: "").
:param mode: File mode, e.g., 'r' or 'rb' (default: 'r').
:param openhook: Optional hook to customize file opening.
:param encoding: File encoding (default: None).
:param errors: Error handling mode (default: None).



The ``Iter`` Class
##################

.. contents::
    :backlinks: entry
    :local:



.. _Iter:


|cool| ``class Iter(Generic[T], Iterator[T])``
**********************************************


This class is what allows chaining. Many of the methods in this class
return an instance of Iter_, which allows further chaining. There
are two exceptions to this: *sources* and *sinks*.

A "source" is usually a ``classmethod`` which can be used as an
initializer to produce data via an iterable. For example, the Iter.range_
classmethod can be used to get a sequence of numbers:

.. code-block:: python

    >>> Iter.range(1_000_000).take(3).collect()
    [0, 1, 2]

Even though our range was a million elements, the iterator chaining
took only 3 of those elements before collecting.

A "sink" is a method that is usually the last component of a processing
chain and often (but not always!) consumes the entire iterator. In the
example above, the call to Iter.collect_ was a sink. Note that we still
call it a sink even though it did not consume the entire iterator.

We're using the term "source" to refer to a classmethod of Iter_ that
produces data; but, the most typical source is going to be data that
you provide. Iter_ can be called with anything that is iterable, including
sequences, iterators, mappings, sets, generators and so on.

Examples:

.. code-block:: python

    List
    >>> Iter([1, 2, 3]).map(lambda x: x * 2).sum()
    12

    Generator
    >>> Iter((1, 2, 3)).map(lambda x: x * 2).sum()
    12
    >>> def g():
    ...     for i in [1, 2, 3]:
    ...         yield i
    >>> Iter(g()).map(lambda x: x * 2).sum()
    12

    Iterator
    >>> Iter(iter([1, 2, 3])).map(lambda x: x * 2).sum()
    12

    Dict
    >>> Iter(dict(a=1, b=2)).map(lambda x: x.upper()).collect()
    ['A', 'B']
    >>> d = dict(a=1, b=2, c=3)
    >>> Iter(d.items()).starmap(lambda k, v: v).map(lambda x: x * 2).sum()
    12

A common error with generators is forgetting to actually evaluate, i.e.,
call a generator function. If you do this there's a friendly error
pointing out the mistake:

.. code-block:: python

    >>> def mygen(): yield 123
    >>> Iter(mygen).collect()
    Traceback (most recent call last):
        ...
    TypeError: It seems you passed a generator function, but you
    probably intended to pass a generator. Remember to evaluate the
    function to obtain a generator instance:
               
    def mygen():
        yield 123
               
    Iter(mygen)    # ERROR - a generator function object is not iterable
    Iter(mygen())  # CORRECT - a generator instance is iterable.
    >>> Iter(mygen()).collect()
    [123]

Instance of Iter_ are resumable. Once an instance it created, it can
be partially iterated in successive calls, like the following example
shows:

.. code-block:: python

    >>> it = Iter.range(1_000_000)
    >>> it.take(3).collect()
    [0, 1, 2]
    >>> it.take(4).collect()
    [3, 4, 5, 6]
    >>> # Consume most of the stream, collect the last few
    >>> it.consume(999_990).collect()
    [999997, 999998, 999999]

This class implements the chaining. However, the module-level functions
in excitertools_, such as range_, zip_ and so on, also return
instances of Iter_, so they allow the chaining to continue. These are
equivalent:

.. code-block:: python

    >>> Iter.range(10).filter(lambda x: x > 7).collect()
    [8, 9]
    >>> range(10).filter(lambda x: x > 7).collect()
    [8, 9]

It is intended that the module-level functions can act as drop-in
replacements for the builtins they wrap:

>>> import builtins
>>> list(builtins.range(3))
[0, 1, 2]
>>> list(range(3))  # This is excitertools.range!
[0, 1, 2]
>>> list(Iter.range(3))
[0, 1, 2]

In your own code where you might like to use the excitertools_ version of
range_ and the other functions, you can just import it and use it to access all the other
cool stuff:

.. code-block:: python

    # mymodule.py
    from excitertools import (
        range,
        map,
        filter,
        reduce,
        repeat,
        count,
        enumerate,
        zip,
        ...
    )

    def func(inputs):
        data = (
            map(lambda x: x + 2, inputs)
                .enumerate()
                .filter(lambda x: x[1] > 10)
                ...
                .collect()

        )

Alternatively, if you don't want to hide the builtins you can do just
fine with importing this class only, or even importing the module only:

.. code-block:: python

    # mymodule.py - same example as before
    import excitertools

    def func(inputs):
        data = (
            excitertools.Iter(inputs)
                .map(lambda x: x + 2, inputs)
                .enumerate()
                .filter(lambda x: x[1] > 10)
                ...
                .collect()
        )

        # Do something with data

There are several valuable additions to the standard *itertools* and
more-itertools_ functions. These usually involve sources and sinks,
which are ways of getting data into an iterator pipeline, and then
getting results out again. In the majority of documentation examples
shown here, the Iter.collect_ method is used to collect all the
remaining data on a stream into a list; but in practice this is not
useful because large lists consume memory.

In practice it is more useful to send iterator data to one of these
common sinks:

- files
- sockets
- queues
- HTTP APIs
- Cloud storage buckets
- (Ideas for more to add here?)

Iter_ has support for these use-cases, both for reading and for writing.



.. _Iter.next:


``Iter.next(self) -> "T"``
==========================
Convenience function to avoid having to wrap an interator with
the `next()` builtin, just to advance it by one step (and return
the value). Typical use cases for this might be for tutorials
and explainers, where you want to show the next value in a
sequence.

.. code-block:: python

    >>> it = Iter(range(5))
    >>> it.next()
    0
    >>> it.next()
    1
    >>> it.collect()
    [2, 3, 4]



.. _Iter.register:


``@classmethod Iter.register(cls, *func)``
==========================================

Add a new method to Iter_. Sure, you could subclass Iter_ to get
new chaining features, but it would be neat to let all existing
Iter_ instance just immediately have the new registered function
available.

The new function must take ``iterable`` as the first parameter.

.. code-block:: python

    >>> def up(iterable):
    ...     for v in iterable:
    ...         yield v.upper()
    >>> Iter.register(up)
    >>> Iter('abc').up().collect()
    ['A', 'B', 'C']
    >>> def poly(iterable, a, b, c):
    ...     # Polynomials a.x^2 + b.x + c
    ...     for x in iterable:
    ...         yield a*x**2 + b*x + c
    >>> Iter.register(poly)
    >>> Iter(range(-5, 5, 1)).poly(1, -5, 6).collect()
    [56, 42, 30, 20, 12, 6, 2, 0, 0, 2]

Here's a math round-trip rollercoaster.

.. code-block:: python

    >>> import math
    >>> def log(iterable):
    ...     for x in iterable:
    ...         yield math.log(x)
    >>> def exp(iterable):
    ...     for x in iterable:
    ...         yield math.exp(x)
    >>> def rnd(iterable):
    ...     for x in iterable:
    ...         yield round(x)
    >>> Iter.register(log, exp, rnd)
    >>> Iter(range(5)).exp().log().rnd().collect()
    [0, 1, 2, 3, 4]

These are silly examples, but hopefully you get the idea.



.. _Iter.collect:


|sink| ``Iter.collect(self, container=list) -> "List[T]"``
==========================================================



This is the most common way of "realizing" an interable chain
into a concrete data structure. It should be the case that this
is where most of the memory allocation occurs.

The default container is a list and you'll see throughout this
documentation that most examples produce lists. However,
any container, and indeed any function, can be used as the sink.

The basic example:

.. code-block:: python

    >>> Iter(range(3)).collect()
    [0, 1, 2]
    >>> Iter(range(3)).collect(tuple)
    (0, 1, 2)

You must pay attention to some things. For example, if your
iterable is a string, the characters of the string are what
get iterated over, and when you collect you'll get a collection
of those atoms. You can however use ``str`` as your "container
function" and that will give you back a string. It's like a join
with blank joiner.

.. code-block:: python

    >>> Iter('abc').collect()
    ['a', 'b', 'c']
    >>> Iter('abc').collect(str)
    'abc'

With some types, things get a little more tricky. Take ``bytes``
for example:

.. code-block:: python

    >>> Iter(b'abc').collect()
    [97, 98, 99]

You probably didn't expect to get the integers back right? Anyhow,
you can use ``bytes`` as the "collection container", just like
we did with strings and that will work:

.. code-block:: python

    >>> Iter(b'abc').collect(bytes)
    b'abc'
    >>> Iter(b'abc').collect(bytearray)
    bytearray(b'abc')

The other standard collections also work, here's a set for
completeness.

.. code-block:: python

    >>> Iter('abcaaaabbbbccc').collect(set) == {'a', 'b', 'c'}
    True

String subclasses also work.

.. code-block:: python

    >>> class MyString(str): pass
    >>> out = Iter(MyString('abc')).collect(MyString)
    >>> out
    'abc'
    >>> type(out)
    <class 'excitertools.MyString'>



.. _Iter.open:


|cool| |source| ``@classmethod Iter.open(cls, file, mode="r", buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None, ) -> "Iter"``
==============================================================================================================================================================




Wrap the ``open()`` builtin precisely, but return an ``Iter``
instance to allow function chaining on the result.

I know you're thinking that we should always use a context
manager for files. Don't worry, there is one being used
internally. When the iterator chain is terminated the underlying
file will be closed.

>>> import tempfile
>>> with tempfile.TemporaryDirectory() as td:
...     # Put some random text into a temporary file
...     with open(td + 'text.txt', 'w') as f:
...         f.writelines(['abc\n', 'def\n', 'ghi\n'])
...
...     # Open the file, filter some lines, collect the result
...     Iter.open(td + 'text.txt').filter(lambda line: 'def' in line).collect()
['def\n']

Note that this is a convenience method for *reading* from a file,
not for writing. The function signature includes the ``mode``
parameter for parity with the builtin ``open()`` function, but
only reading is supported.



.. _Iter.read_lines:


|source| ``@classmethod Iter.read_lines(cls, stream: IO[str], rewind=True)``
============================================================================



Read lines from a file-like object.

First, let's put some data in a file. We'll be using that
file in the examples that follow.

.. code-block:: python

    >>> import tempfile
    >>> td = tempfile.TemporaryDirectory()
    ... # Put some random text into a temporary file
    >>> with open(td.name + 'text.txt', 'w') as f:
    ...     f.writelines(['abc\n', 'def\n', 'ghi\n'])
    ...

Use read_lines to process the file data

.. code-block:: python

    >>> with open(td.name + 'text.txt') as f:
    ...     Iter.read_lines(f).filter(lambda line: 'def' in line).collect()
    ['def\n']

The ``rewind`` parameter can be used to read sections of a file.

.. code-block:: python

    >>> with open(td.name + 'text.txt') as f:
    ...     part1 = Iter.read_lines(f).take(1).collect()
    ...     part2 = Iter.read_lines(f, rewind=False).collect()
    >>> part1
    ['abc\n']
    >>> part2
    ['def\n', 'ghi\n']
    >>> td.cleanup()



.. _Iter.read_bytes:


|source| ``@classmethod Iter.read_bytes(cls, stream: IO[bytes], size: Union[Callable[[], int], int] = -1, rewind=True)``
========================================================================================================================



The ``size`` parameter can be used to control how many bytes are
read for each advancement of the iterator chain. Here we set ``size=1``
which means we'll get back one byte at a time.

.. code-block:: python

    >>> import tempfile
    >>> td = tempfile.TemporaryDirectory()
    >>> filename = td.name + 'bytes.bin'

Put some random text into a temporary file:

.. code-block:: python

    >>> with open(filename, 'wb') as f:
    ...     x = f.write(b'\x00' * 100)
    ...
    >>> with open(filename, 'rb') as f:
    ...     data = Iter.read_bytes(f, size=1).collect()
    ...     len(data)
    100
    >>> with open(filename, 'rb') as f:
    ...     data = Iter.read_bytes(f).collect()
    ...     len(data)
    1

A little more ambitious. Because ``size`` is a callable, we can use
a ``deque`` and a ``side_effect`` to pass information back into
the reader to control how many bytes are read in each chunk.

In this example we're reading 1 byte at a time. In a real example
you might have a sequence of headers and bodies, where headers
give size information about how many bytes are in the body
corresponding to that header. Then you can precisely read
each body in sequence.

.. code-block:: python

    >>> from collections import deque
    >>> read_sizes = deque([1])
    >>> with open(filename, 'rb') as f:
    ...     data = (
    ...         Iter
    ...             .read_bytes(f, size=lambda: read_sizes.popleft())
    ...             .side_effect(lambda bytes: read_sizes.append(1))
    ...             .collect()
    ...     )
    ...     len(data)
    100

The ``rewind`` parameter can be used to read sections of a file.

.. code-block:: python

    >>> with open(filename, 'rb') as f:
    ...     part1 = Iter.read_bytes(f, size=10).take(1).collect()
    ...     part2 = Iter.read_bytes(f, rewind=False).collect()
    >>> part1
    [b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
    >>> len(part2[0])
    90
    >>> td.cleanup()



.. _Iter.write_text_to_stream:


|sink| ``Iter.write_text_to_stream(self, stream: IO[str], insert_newlines=True, flush=True)``
=============================================================================================



.. code-block:: python

    >>> import tempfile
    >>> td = tempfile.TemporaryDirectory()
    >>> filename = td.name + 'text.txt'

    >>> data = ['a', 'b', 'c']
    >>> with open(filename, 'w') as f:
    ...     Iter(data).map(str.upper).write_text_to_stream(f)
    ...     with open(filename) as f2:
    ...         Iter.read_lines(f2).concat()
    'A\nB\nC'

If some prior step adds newlines, or more commonly, newlines
originate with a data source and are simply carried through the
processing chain unaltered, disable the insertion of newlines:

.. code-block:: python

    >>> with open(filename, 'w') as f:
    ...     Iter(data).map(str.upper).write_text_to_stream(f, insert_newlines=False)
    ...     with open(filename) as f2:
    ...         Iter.read_lines(f2).concat()
    'ABC'

Multiple successive writes may be slowed down by the default
``flush=True`` parameter. In this case you can delay flushing until
everything has been written.

.. code-block:: python

    >>> with open(filename, 'w') as f:
    ...     Iter(data).map(str.upper).write_text_to_stream(f, flush=False)
    ...     Iter(data).map(str.upper).write_text_to_strea