semvec/pypi-packages huggingface dataset:
fsdata 0.0.6
Simple data access layer over fsspec
Requires Python: >=3.10
Keywords: data-access, pathlib, fsspec

# Simple data catalog library for python

This project is a trivial attempt at offering basic catalog functionality for structured datasets stored in local or remote folders. The library uses `universal_pathlib` to access remote storage locations like S3, Google Cloud Storage, etc ... The library reads a config file called `fsdata.ini` which defines a list of collections, one per section. Each collection corresponds to a local or remote folder containing a collection of files, all with the same format and extension. Currently, as a prototype, the library supports only pandas dataframes saved as `.parquet` files. The library uses local caching to avoid fetching the same data multiple times.

> **Warning** This project is for exploration only.

## Configuration

The configuration file `fsdata.ini` has one section for each collection, with the section name for name and with a `path` key pointing to its location.
The config file should be located in the the standard XDG config directory `XDG_CONFIG_HOME` (or ~/.config).

```ini
# fsdata.ini
[samples]
path = s3://my-bucket/samples

[datasets]
path = s3://my-bucket/datasets

[testdata]
path = s3://my-bucket/testdata
```


## Usage

To access a given collection just use the `collection` method.

```python
import fsdata

samples = fsdata.collection("samples")
```

To list items in a collections

```python
samples.items()
```

Please note that item names are bare names without extension.

To load data use the `load` method.

```python
samples.load("my-sample")
```

To save data use the `save` method.

```python
samples.save("my-sample", data)
```

You can also load on item directly with `fsdata.load` method

```python
fsdata.load("samples", "my-sample")
```

## Installation

You can install the package with `pip`

```shell
pip install fsdata
```

You can also specify any of the extra dependencies `s3`, `gcs`, `adl`

```shell
pip install "fsdata[s3]"
```

## Requirements

- pandas
- pyarrow
- universal_pathlib
- fsspec backends like s3fs, etc ... as applicable


## Related Projects and Resources
- [intake](https://github.com/intake/intake) - Lightweight package for finding, investigating, loading and disseminating data.
- [quilt](https://github.com/quiltdata/quilt) - Quilt is a data mesh for connecting people with actionable data
- [pystore](https://github.com/ranaroussi/pystore) - Fast data store for Pandas time-series data
- [pandas](https://github.com/pandas-dev/pandas) - Flexible and powerful data analysis / manipulation library for Python
- [pyarrow](https://github.com/apache/arrow) - Universal columnar format and multi-language toolbox
- [parquet](https://github.com/apache/parquet-format) - Apache Parquet Format
- [fsspec](https://github.com/fsspec/filesystem_spec) - Filesystem interfaces for Python
- [universal_pathlib](https://github.com/fsspec/universal_pathlib) - pathlib api extended to use fsspec backends