semvec/pypi-packages huggingface dataset:
luigi-daisy 0.0.3
Utility wrapper of luigi
License: BSD

# daisy

Thin wrapper of [luigi](https://github.com/spotify/luigi) for utility.

# Features

## Formatted local targets

Classes inheriting `daisy.FormattedLocalTargetBase` is provided
for dumping and loading objects by one-liner.

``` python
import daisy
import pandas as pd

df = pd.DataFrame({"a": [1,2,3], "b": [4,5,6]})
df
# =>
#    a  b
# 0  1  4
# 1  2  5
# 2  3  6

targ = daisy.CsvTarget("./output.csv")

# dumping
targ.dump(df)

# loading
df2 = targ.load()

df2
# =>
#    a  b
# 0  1  4
# 1  2  5
# 2  3  6

# `daisy.FormattedLocalTargetBase` also inherits `luigi.LocalTaget`
# so that original api is also enabled.
with targ.open("r") as fd:
    s = fd.read()
```

## Default output for task

`daisy.Task` inherits `luigi.Task` and provides default `output` features.

By setting file extension via `ext` attribute,
daisy automatically configure corresponding `FormattedLocalTarget` with default file name.


### Single output

``` python
class TaskA(daisy.Task):
    param1 = daisy.Parameter()
    ext = "csv"
```

is equivalent to:

``` python
class TaskA(luigi.Task):
    param1 = luigi.Parameter()

    def output(self):
        return daisy.CsvTarget("./data/TaskA/TaskA(param1={}).csv".format(self.param1))
```


### Multiple outputs

``` python
class TaskA(daisy.Task):
    param1 = daisy.Parameter()

    ext = {
        "vectors": "npy",
        "metadata": "json"
    }
```

is equivalent to:

``` python
class TaskA(luigi.Task):
    param1 = luigi.Parameter()

    def output(self):
        return {
            "vectors": daisy.NpyTarget("./data/TaskA/TaskA(param1={}).npy".format(self.param1)),
            "metadata": daisy.JsonTarget("./data/TaskA/TaskA(param1={}).json".format(self.param1))
            }
```

Available extension and file types are as follows:

| Target class | Object type | extension |
| --- | --- | --- |
| `CsvTarget` | `pandas.DataFrame` | `csv` |
| `NpyTarget` | `numpy.ndarray` | `npy` |
| `JsonTarget` | `dict` | `json` |
| `PickleTarget` | `object` | `pkl` `pickle` |
| `FeatherTarget` | `pandas.DataFrame` | `feather` |

view [source code](./daisy/formatted_target.py) for detail.


## Configuration

For configuration, edit `[daisy]` section of `luigi.cfg`.

``` INI
[daisy]

# default output directory
data_dir = luigi.Parameter("./data")
```