necstdb 0.2.9
Database for NECST.
License: MIT
Requires Python: >=3.6.1,<4.0.0

# NECSTDB

[![PyPI](https://img.shields.io/pypi/v/necstdb.svg?label=PyPI&style=flat-square)](https://pypi.org/pypi/necstdb/)
[![Python](https://img.shields.io/pypi/pyversions/necstdb.svg?label=Python&color=yellow&style=flat-square)](https://pypi.org/pypi/necstdb/)
[![Test](https://img.shields.io/github/workflow/status/ogawa-ros/necstdb/Test?logo=github&label=Test&style=flat-square)](https://github.com/ogawa-ros/necstdb/actions)
[![License](https://img.shields.io/badge/license-MIT-blue.svg?label=License&style=flat-square)](LICENSE)

Database for NECST.

## TL;DR

NECST, an abbreviation of *NEw Control System for Telescope*, is a flexible control system for radio telescopes. Its efficient data storage is provided here.

The database contains tables, which keep individual topic of time series data with some metadata attached to them, e.g. spectral data from one spectrometer board (array of data + timestamp), weather data (temperature + humidity + wind speed + ... + timestamp), etc.

## Features

This package provides:

- database writer with quite flexible data format support
- database reader which supports various output format

## Installation

```shell
pip install necstdb
```

## Usage

### Database Writer

1. Create new database instance

    ```python
    >>> db_dir = "path/to/new_database_directory"
    >>> db = necstdb.opendb(db_dir, mode="w")  # "w" stands for "write"
    ```

2. Create table with data information

    ```python
    >>> data1_info = {
    ...     "data": [
    ...         {"key": "recorded_time", "format": "d", "size": 8},
    ...         {"key": "n_scan_lines", "format": "i", "size": 4},
    ...         {"key": "obsmode", "format": "3s", "size": 3},
    ...         {"key": "timestamp", "format": "d", "size": 8},
    ...     ],
    ...     "memo": "generated by db_logger_operation",
    ...     "necstdb_version": necstdb.__version__,
    ... }
    >>> db.create_table("data1", data1_info)
    ```

    The keys "key" and "format" in "data" list are required. The format characters and the sizes are listed below. For more information, see the struct module [documentation](https://docs.python.org/3/library/struct.html#format-characters) and/or the ROS message [wiki](http://wiki.ros.org/msg).

    - *Changed in v0.2.5: "size" in "data" list is no longer required, but is optional.*

    The "memo" and "necstdb_version" keys are not necessary. You can also add any other keys to the data information dict.

    | ROS format | Format character | Size \[byte\] |
    | ---------- | ---------------- | ------------- |
    | bool       | ``?``            | 1             |
    | int8       | ``b``            | 1             |
    | int16      | ``h``            | 2             |
    | int32      | ``i``            | 4             |
    | int64      | ``q``            | 8             |
    | uint8      | ``B``            | 1             |
    | uint16     | ``H``            | 2             |
    | uint32     | ``I``            | 4             |
    | uint64     | ``Q``            | 8             |
    | float32    | ``f``            | 4             |
    | float64    | ``d``            | 8             |
    | string     | ``[length]s``    | 1 * length    |

    ***NOTE:***
    Array data are also supported. To write them, use repeat count syntax (e.g. ``3d``) or format character sequence syntax (e.g. ``ddd``) for the "format", and the sum of the elements' sizes for the "size".

    ```python
    {"key": "data_array", "format": "3d", "size": 3 * 8}  # repeat count syntax
    {"key": "data_array", "format": "ddd", "size": 8 + 8 + 8}  # format character sequence syntax
    ```

3. Write data into table

    ```python
    >>> table1 = db.open_table("data1", mode="ab")  # "ab" stands for "append binary"
    >>> data1 = [1.6294488758e9, 3, b"SKY", 1.6294488757e9]  # string data are not allowed, use bytes instead
    >>> table1.append(*data1)
    ```

    Call the ``append`` method every time you get new data.

    ***NOTE:***
    Data to pass to ``append`` method should be flattened. The nested structure will be reconstructed on reading.

    ```python
    data = [1, 2, [3, 4, 5], 6, 7]
    data = necstdb.utils.flatten_data(data)
    table.append(*data)
    ```

    ***NOTE:***
    Multi-dimensional data are not supported.

    ***NOTE:***
    Shape of array or length of string in every data must be the same as the one in data information. (Mismatch of string length won't raise an error, but stored data can be broken or incomplete.)

4. Save files into the database

    ```python
    >>> db.save_file("example.txt", "Content of the file.")
    ```

### Database Reader

1. Open the database instance

    ```python
    >>> db = necstdb.opendb("path/to/database_directory")
    ```

2. Read a desired topic

    ```python
    >>> data1 = db.open_table("data1").read(astype="array")
    >>> data1
    array([[1.6294488758e9, 3, b'SKY', 1.6294488757e9], ...], dtype=[('recorded_time', '<f8'), ('n_scan_lines', '<i4'), ('obsmode', '|S3'), ('timestamp', '<f8')])
    ```

    The supported ``astype`` keywords for ``read`` method are:

    | Output type      | Keywords                                             | Notes |
    | ---------------- | ---------------------------------------------------- | ----- |
    | tuple            | "tuple"                                              | \*1   |
    | dict             | "dict"                                               |       |
    | pandas.DataFrame | "pandas", "dataframe", "data_frame", "df"            | \*2   |
    | numpy.ndarray    | "array", "structuredarray", "structured_array", "sa" |       |
    | bytes            | "buffer", "raw"                                      |       |

    ***\*1***: Array data are not supported, but will be flattened.  
    ***\*2***: *Changed in v0.2.5: Keyword ``df`` is now supported.*

3. Read files saved in the database

    ```python
    >>> db.read_file("example.txt")
    "Content of the file.", ""
    ```

    This method assumes the file can be read as `str`. This attempt will fail for binary files, so use `asbytes` option explicitly.

    ```python
    >>> db.read_file("example_binary.data", asbytes=True)
    b"Content of the file.", b""
    ```

    This method returns 2 `str` or `bytes` values. The second value is metadata for the file saved using `db.save_file(filename, data, metadata)` method.

### Misc

#### List all the tables contained in the database

```python
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.list_tables()
['data1', 'spectral_data', 'weather_data', ...]
```

#### Archive the database

```python
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.checkout(saveto="path/to/archive.tar.gz", compression="gz")
```

#### Get informations of all tables in the database

```python
>>> db = necstdb.opendb("path/to/database_directory")
>>> db.get_info()
"""
                    file size [byte]    #records    record size [byte]  format
table name
data1               2448                102         24                  di3sd
spectral_data       41948160            320         131088              d32768fd
weather_data        6328                113         56                  ddddddd
"""
```

#### Read particular columns and/or rows of the database

```python
>>> db = necstdb.opendb("path/to/database_directory")
>>> data = db.open_table("data1").read(num=5, start=3, cols=["timestamp", "obsmode"], astype="tuple")  # order of cols won't be preserved
((b'SKY', 1.6294488788e9)  # 4th element (caution 0-based indexing)
 (b'SKY', 1.6294488798e9)
 (b'SKY', 1.6294488808e9)
 (b'SKY', 1.6294488818e9)
 (b'HOT', 1.6294488828e9))
```

#### Flatten nested array

```python
>>> data = [1, 2, 3, [4, 5], 6]
>>> flattened = necstdb.utils.flatten_data(data)
[1, 2, 3, 4, 5, 6]
```