morefs 0.2.2
A collection of self-contained fsspec-based filesystems
License: Apache-2.0
Requires Python: >=3.8

morefs
======

|PyPI| |Status| |Python Version| |License|

|Tests| |Codecov| |pre-commit| |Black|

.. |PyPI| image:: https://img.shields.io/pypi/v/morefs.svg
   :target: https://pypi.org/project/morefs/
   :alt: PyPI
.. |Status| image:: https://img.shields.io/pypi/status/morefs.svg
   :target: https://pypi.org/project/morefs/
   :alt: Status
.. |Python Version| image:: https://img.shields.io/pypi/pyversions/morefs
   :target: https://pypi.org/project/morefs
   :alt: Python Version
.. |License| image:: https://img.shields.io/pypi/l/morefs
   :target: https://opensource.org/licenses/Apache-2.0
   :alt: License
.. |Tests| image:: https://github.com/iterative/morefs/workflows/Tests/badge.svg
   :target: https://github.com/iterative/morefs/actions?workflow=Tests
   :alt: Tests
.. |Codecov| image:: https://codecov.io/gh/iterative/morefs/branch/main/graph/badge.svg
   :target: https://app.codecov.io/gh/iterative/morefs
   :alt: Codecov
.. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white
   :target: https://github.com/pre-commit/pre-commit
   :alt: pre-commit
.. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black
   :alt: Black


Features
--------

*morefs* provides standalone fsspec-based filesystems like:

* ``AsyncLocalFileSystem`` that provides async implementation of ``LocalFileSystem``.
* In-memory filesystems ``DictFileSystem`` built on nested dictionaries and ``MemFS`` built on tries, and are much faster than fsspec's ``MemoryFileSystem``.
* ``OverlayFileSystem`` that allows to overlay multiple fsspec-based filesystems.

Installation
------------

You can install *morefs* via pip_ from PyPI_:

.. code:: console

   $ pip install morefs

You might need to install with extras for some filesystems:

.. code:: console

   $ pip install morefs[asynclocal]  # for installing aiofile dependency for AsyncLocalFileSystem
   $ pip install morefs[memfs]  # for installing pygtrie dependency for MemFS


Usage
-----

AsyncLocalFileSystem
~~~~~~~~~~~~~~~~~~~~

Extended version of ``LocalFileSystem`` that also provides async methods.

.. code:: python

    import asyncio
    from morefs.asyn_local import AsyncLocalFileSystem

    async def main():
        fs = AsyncLocalFileSystem(auto_mkdir=False)

        f = await fs.open_async("foo", mode="w")
        async with f:
            await f.write("foobar")

        content = await fs._cat("foo")
        print(content)
        print(fs.cat("foo"))  # you can still use sync methods

    asyncio.run(main())


DictFS
~~~~~~

DictFS is a nested dictionary-based, in-memory filesystem
and acts more like a real LocalFileSystem.

.. code:: python

    from morefs.dict import filesystem

    fs = DictFS()


MemFS
~~~~~

MemFS is a trie-based in-memory filesystem, and acts like a bucket storage.

.. code:: python

    from morefs.memory import MemFS

    fs = MemFS()


OverlayFileSystem
~~~~~~~~~~~~~~~~~

.. code:: python

    from morefs.overlay import OverlayFileSystem

    # use localfilesystem for write, overlay all filesystems for read
    fs = OverlayFileSystem(file={"auto_mkdir": True}, s3={"anon": True})
    # or you can pass filesystem instances directly
    # as variable positional arguments or with keyword argument `filesystems=[]`
    fs = OverlayFileSystem(LocalFileSystem(), s3={"anon": True})


Contributing
------------

Contributions are very welcome.
To learn more, see the `Contributor Guide`_.


License
-------

Distributed under the terms of the `Apache 2.0 license`_,
*morefs* is free and open source software.


Issues
------

If you encounter any problems,
please `file an issue`_ along with a detailed description.


.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0
.. _PyPI: https://pypi.org/
.. _file an issue: https://github.com/iterative/morefs/issues
.. _pip: https://pip.pypa.io/
.. github-only
.. _Contributor Guide: CONTRIBUTING.rst