helikite-data-processing 1.1.3
Library to generate quicklooks and data quality checks on Helikite campaigns
Requires Python: <3.13,>=3.11

# helikite-data-processing

This library supports Helikite campaigns by unifying field-collected data, generating quicklooks, and performing quality control on instrument recordings. It is now available on PyPI, can be used via a command‐line interface (CLI), and also runs in Docker containers if needed.

## Table of Contents
1. [Getting Started](#getting-started)
   1. [Pip Installation](#pip-installation)
   2. [Docker](#docker)
   3. [Makefile](#makefile)
2. [Using the Library](#using-the-library)
3. [Cleaner](#cleaner)
4. [Documentation & Examples](#documentation--examples)
5. [Command-line Usage](#command-line-usage)
6. [Development](#development)
   1. [The Instrument class](#the-instrument-class)
   2. [Adding more instruments](#adding-more-instruments)
7. [Configuration](#configuration)
   1. [Application constants](#application-constants)
   2. [Runtime configuration](#runtime)


# Getting Started

## Pip Installation

Helikite is published on [PyPI](https://pypi.org/project/helikite-data-processing/). To install it via `pip`, run:

```bash
pip install helikite-data-processing
```

After installation, the CLI is available as a system command:

```bash
helikite --help
```

## Docker

> **Note:** Docker usage is now optional. For most users, installing via pip is the recommended approach.

### Building and Running with Docker

1. **Build the Docker image:**

   ```bash
   docker build -t helikite .
   ```

2. **Generate project folders and create the configuration file:**

   ```bash
   docker run \
       -v ./inputs:/app/inputs \
       -v ./outputs:/app/outputs \
       helikite:latest generate_config
   ```

3. **Preprocess the configuration file:**

   ```bash
   docker run \
       -v ./inputs:/app/inputs \
       -v ./outputs:/app/outputs \
       helikite:latest preprocess
   ```

4. **Process data and generate plots:**

   ```bash
   docker run \
       -v ./inputs:/app/inputs \
       -v ./outputs:/app/outputs \
       helikite:latest
   ```

You can also use the pre-built image from GitHub Packages:

```bash
docker run \
   -v ./inputs:/app/inputs \
   -v ./outputs:/app/outputs \
   ghcr.io/eerl-epfl/helikite-data-processing:latest generate_config
```

## Makefile

The Makefile provides simple commands for common tasks:

```bash
make build             # Build the Docker image
make generate_config   # Generate the configuration file in the inputs folder
make preprocess        # Preprocess data and update the configuration file
make process           # Process data and generate plots (output goes into a timestamped folder)
```

# Using the Library

Helikite can be used both as a standalone CLI tool and as an importable Python package.
For non-programmers, the CLI is the simplest way to use the library.
For programmers, the library can be imported and used in your own scripts:

```python
import helikite
from helikite.processing import preprocess, sorting
from helikite.constants import constants

# For example, to generate a configuration file programmatically:
preprocess.generate_config()
```

A complete list of available functions and modules is documented on the [auto-published documentation site](https://eerl-epfl.github.io/helikite-data-processing/).

# Cleaner

The `cleaner` module is designed to tidy up output folders generated by the application. For instructions on how to use it, refer to the [Level 0 notebook](./notebooks/level0.ipynb).

# Documentation & Examples

For full API documentation, usage examples, and tutorials, please visit the [Helikite Data Processing Documentation](https://eerl-epfl.github.io/helikite-data-processing/).

The `notebooks` folder also contains a [Level 0 processing example](./notebooks/level0.ipynb) that demonstrates how to use the library for basic data processing tasks.

# Command-line Usage

Once installed (via pip or Docker), you can use the CLI to run the three main stages of the application:

1. **Generate a configuration file:**
   This creates a config file in your `inputs` folder.
   ```bash
   helikite generate-config
   ```

2. **Preprocess:**
   Scans the input folder, associates raw instrument files to configurations, and updates the config file.
   ```bash
   helikite preprocess
   ```

3. **Process:**
   Processes the input data based on the configuration, normalizes timestamps, and generates plots.
   (Running without any command runs this stage.)
   ```bash
   helikite
   ```

For detailed help on any command, append `--help` (e.g., `helikite preprocess --help`).

# Development

## The Instrument class

The structure of the Instrument class allows specific data cleaning activities to be overridden for each instrument that inherits from it. The main application (in `helikite.py`) calls these class methods to process the data.

## Adding more instruments

The configuration file is generated during the `generate_config`/`preprocess` steps by iterating over the instantiated classes imported in `helikite/instruments/__init__.py`. To add a new instrument, create a subclass of `Instrument` and import it in `__init__.py`.

Firstly, the class should inherit from `Instrument` and set a unique name (e.g., for the `MCPC` instrument):

```python
def __init__(self, *args, **kwargs) -> None:
    super().__init__(*args, **kwargs)
    self.name = 'mcpc'
```

The minimum functions required are:

- `file_identifier()`: Accepts the first 50 lines of a CSV file and returns `True` if it matches the instrument’s criteria (typically checking header content).

  ```python
  # Example for the pico instrument:
  def file_identifier(self, first_lines_of_csv) -> bool:
      if ("win0Fit0,win0Fit1,win0Fit2,win0Fit3,win0Fit4,win0Fit5,win0Fit6,"
          "win0Fit7,win0Fit8,win0Fit9,win1Fit0,win1Fit1,win1Fit2") in first_lines_of_csv[0]:
          return True
      return False
  ```

- `set_time_as_index()`: Converts the instrument's timestamp information into a common pandas `DateTimeIndex`.

  ```python
  # Example for the filter instrument:
  def set_time_as_index(self, df: pd.DataFrame) -> pd.DataFrame:
      df['DateTime'] = pd.to_datetime(
          df['#YY/MM/DD'].str.strip() + ' ' + df['HR:MN:SC'].str.strip(),
          format='%y/%m/%d %H:%M:%S'
      )
      df.drop(columns=["#YY/MM/DD", "HR:MN:SC"], inplace=True)
      df.set_index('DateTime', inplace=True)
      return df
  ```

For more details and examples, refer to the [auto-published documentation](https://eerl-epfl.github.io/helikite-data-processing/).

# Configuration

There are three sources of configuration parameters:

## Application constants

These are defined in `helikite/constants.py` and include settings such as filenames, folder paths for inputs/outputs, logging formats, and default plotting parameters.

## Runtime configuration

The runtime configuration is stored in `config.yaml` (located in your `inputs` folder). This file is generated during the `generate_config` or `preprocess` steps. It holds runtime arguments for each instrument (e.g., file locations, time adjustments, and plotting settings).

Below is an example snippet from a generated `config.yaml`:

```yaml
global:
  time_trim:
    start: 2022-09-29 10:21:58
    end: 2022-09-29 12:34:36
ground_station:
  altitude: null
  pressure: null
  temperature: 7.8
instruments:
  filter:
    config: filter
    date: null
    file: /app/inputs/220209A3.TXT
    pressure_offset: null
    time_offset:
      hour: 5555
      minute: 0
      second: 0
plots:
  altitude_ground_level: false
  grid:
    resample_seconds: 60
```