wq.io 1.1.0
Consistent iterable API for reading and writing to external datasets
License: MIT

|wq.io|

`wq.io <https://wq.io/wq.io>`__ is a Pythonic library for consuming
(input), iterating over, and generating (output) external data resources
in various formats. wq.io facilitates interoperability between the `wq
framework <https://wq.io/>`__ and other systems and formats.

wq.io is `designed to be customized <https://wq.io/docs/custom-io>`__,
with a `base class <https://wq.io/docs/base-io>`__ and modular mixin
classes that handle `loading <https://wq.io/docs/loaders>`__,
`parsing <https://wq.io/docs/parsers>`__, and
`mapping <https://wq.io/docs/mappers>`__ external data to a convenient
API.

|Latest PyPI Release| |Release Notes| |Documentation| |License| |GitHub
Stars| |GitHub Forks| |GitHub Issues|

|Travis Build Status| |Python Support|

    Somewhat coincidentally, https://wq.io is also the URL for the
    website describing the wq framework as a whole. The documentation
    for wq.io (the library) is available on wq.io (the website) at
    https://wq.io/wq.io.

Getting Started
---------------

.. code:: bash

    # Recommended: create virtual environment
    # python3 -m venv venv
    # . venv/bin/activate

    # Install entire wq suite (recommended)
    pip install wq

    # Install only wq.io
    pip install wq.io

    # To enable wq.io's GIS support
    pip install geopandas # includes Shapely & Fiona

    # To enable wq.io's Excel write support
    pip install xlwt # xls support
    pip install xlsxwriter # xlsx support
    # (xls/xlsx read support is enabled by default)

See `the wq documentation <https://wq.io/docs/>`__ for more information.

Features
--------

wq.io provides a general purpose API for loading, iterating over, and
writing tabular datasets. The basic idea is to avoid needing to remember
the unique usage of e.g.
`csv <https://docs.python.org/3/library/csv.html>`__,
`xlrd <http://www.python-excel.org/>`__, or
`xml.etree <https://docs.python.org/3/library/xml.etree.elementtree.html>`__
every time one needs to work with external data. Instead, wq.io
abstracts these libraries into a consistent interface that works as an
`iterable <https://docs.python.org/3/glossary.html#term-iterable>`__ of
`namedtuples <https://docs.python.org/3/library/collections.html#collections.namedtuple>`__.
Whenever possible, the field names for a dataset are automatically
determined from the source file, e.g. the column headers in an Excel
spreadsheet.

.. code:: python

    from wq.io import ExcelFileIO
    data = ExcelFileIO(filename='example.xls')
    for row in data:
        print(row.name, row.date)

wq.io provides a number of built-in classes like the above, including a
``CsvFileIO``, ``XmlFileIO``, and ``JsonFileIO``. There is also a
convenience function, ``load_file()``, that attempts to automatically
determine which class to use for a given file.

.. code:: python

    from wq.io import load_file
    data = load_file('example.csv')
    for row in data:
        print(row.name, row.date)

All of the included ``*FileIO`` classes support both reading and writing
to external files, though write support for Excel files requires
additional libraries (`xlwt <http://www.python-excel.org/>`__ and
`xlsxwriter <https://xlsxwriter.readthedocs.org/>`__) that aren't listed
as dependencies.

Network Client
~~~~~~~~~~~~~~

wq.io also provides network-capable equivalents of each of the above
classes, to facilitate loading data from third party webservices.

.. code:: python

    from wq.io import JsonNetIO
    class WebServiceIO(JsonNetIO):
        url = "http://example.com/api"
        
    data = WebServiceIO(params={'type': 'all'})
    for row in data:
        print(row.timestamp, row.value)

The powerful `requests <http://python-requests.org/>`__ library is used
internally to load data over HTTP.

Pandas Analysis
~~~~~~~~~~~~~~~

When `Pandas <http://pandas.pydata.org/>`__ is installed, the
``as_dataframe()`` method on wq.io classes can be used to create a
`DataFrame <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html>`__,
enabling more extensive analysis possibilities.

.. code:: python

    instance = WebServiceIO(params={'type': 'all'})
    df = instance.as_dataframe()
    print(df.value.mean())

GIS Support
~~~~~~~~~~~

When `Fiona <https://github.com/Toblerity/Fiona>`__ and
`Shapely <https://github.com/Toblerity/Shapely>`__ are installed, wq.io
can also open and create shapefiles and other OGR-compatible geographic
data formats.

.. code:: python

    from wq.io import ShapeIO
    data = ShapeIO(filename='sites.shp')
    for id, site in data.items():
        print(id, site.geometry.wkt)

Extending wq.io
~~~~~~~~~~~~~~~

Each ``IO`` class is composed of mixin classes
(`loaders <https://wq.io/docs/loaders>`__,
`parsers <https://wq.io/docs/parsers>`__, and
`mappers <https://wq.io/docs/mappers>`__) that handle the various steps
of the process. By extending these mixin or the pre-mixed classes above,
it is straightforward to `extend wq.io <https://wq.io/docs/custom-io>`__
to support arbitrary formats. The `climata
library <https://github.com/heigeo/climata>`__ provides a number of
examples of custom ``IO`` classes for loading climate and hydrology
data.

.. |wq.io| image:: https://raw.github.com/wq/wq/master/images/256/wq.io.png
   :target: https://wq.io/wq.io
.. |Latest PyPI Release| image:: https://img.shields.io/pypi/v/wq.io.svg
   :target: https://pypi.org/project/wq.io
.. |Release Notes| image:: https://img.shields.io/github/release/wq/wq.io.svg
   :target: https://github.com/wq/wq.io/releases
.. |Documentation| image:: https://img.shields.io/badge/Docs-1.1-blue.svg
   :target: https://wq.io/wq.io
.. |License| image:: https://img.shields.io/pypi/l/wq.io.svg
   :target: https://wq.io/license
.. |GitHub Stars| image:: https://img.shields.io/github/stars/wq/wq.io.svg
   :target: https://github.com/wq/wq.io/stargazers
.. |GitHub Forks| image:: https://img.shields.io/github/forks/wq/wq.io.svg
   :target: https://github.com/wq/wq.io/network
.. |GitHub Issues| image:: https://img.shields.io/github/issues/wq/wq.io.svg
   :target: https://github.com/wq/wq.io/issues
.. |Travis Build Status| image:: https://img.shields.io/travis/wq/wq.io.svg
   :target: https://travis-ci.org/wq/wq.io
.. |Python Support| image:: https://img.shields.io/pypi/pyversions/wq.io.svg
   :target: https://pypi.python.org/pypi/wq.io