flexprep 0.1.3
A Python library for pre-processing FLEXPART inputs
Requires Python: <3.13,>=3.10

# flexprep

<p align="center">
    <a href="https://pypi.org/project/flexprep/">
    <img src="https://img.shields.io/pypi/v/flexprep.svg?color=ff69b4" alt="PyPI version">
    </a>
    <a href="https://opensource.org/licenses/mit">
    <img src="https://img.shields.io/badge/licence-MIT-blue.svg" alt="Licence">
    </a>
</p>

> **WARNING:**
>
> This project is in BETA and under active development. Interfaces and functionality are subject to change.

**flexprep** is a Python toolkit to retrieve and preprocess ECMWF IFS fields for the Lagrangian particle dispersion model [**FLEXPART**](https://www.flexpart.eu/).

It uses ECMWF’s [earthkit-data](https://earthkit-data.readthedocs.io/en/latest/) library to load meteorological GRIB data into familiar xarray/numpy arrays for computation, preserves GRIB metadata, and can write the results back to GRIB2.


## Compatibility

Target model: [**FLEXPART V10**](https://www.flexpart.eu/)


## Install the latest release from PyPI

```bash
# pip
pip install flexprep

# or poetry
poetry add flexprep
```


## At a glance

1. **Retrieve IFS forecasts**
   - via [MARS](https://confluence.ecmwf.int/display/UDOC/MARS+user+documentation)
   - via [Polytope](https://polytope.readthedocs.io/en/latest/)
   - from local GRIB files

2. **Preprocess for FLEXPART V10**
   - De-accumulates precipitation, radiation, and fluxes
   - Converts `etadot` → `omega` (Pa s⁻¹)

3. **Write GRIB2**
   - Preserves/overrides GRIB metadata as needed
   - Ensures all fields are written as **GRIB2**

## Documentation

See the `docs/` folder for detailed information:
- [Inputs](docs/inputs.md)
- [Data Sources](docs/sources.md)
- [Preprocessing](docs/preprocessing.md)
- [GRIB I/O](docs/io_grib.md)


## Changelog

See [CHANGELOG.md](CHANGELOG.md) for version history.

## Running Tests

To run the full test suite:
```
git clone https://gitlab.phaidra.org/flexpart/flexprep.git
cd flexprep
poetry install --with dev
poetry run pytest
```

## Feedback

Found a bug or have feedback?
Please [open an issue](https://gitlab.phaidra.org/flexpart/flexprep/-/issues).

## Roadmap / TODO

- [ ] Publish a hosted documentation site (e.g., Read the Docs or GitHub Pages)
- [ ] Add Copernicus Data Store (CDS) as a data source
- [ ] Find suitable package name (flexprep?)


## Testing the flexprep Package

This guide explains how to test the **flexprep** pipeline using example scripts included in the repository.
Each example demonstrates a full workflow: **loading**, **preprocessing**, and **writing** FLEXPART-ready GRIB2 files.


### 1️. Install the package

#### Option A — From PyPI
```bash
pip install flexprep==0.1.3
```

### 2️. Get the example scripts
Clone the repository to access the example workflows.

```bash
git clone https://gitlab.phaidra.org/flexpart/flexprep.git
cd flexprep
```
### 3. Run an example pipeline

Each example runs the flexprep pipeline end-to-end.

| Data source | Command | Notes |
|--------------|----------|-------|
| 🧪 **Bundled test data** | `python examples/run_pipeline_local.py tests/data/test_sources_raw_ifs.tar.gz` | Works offline using a small bundled dataset |
| 💾 **Local GRIB files** | `python examples/run_pipeline_local.py /path/to/grib_dir` | Use your own GRIB directory |
| ☁️ **ECMWF MARS archive** | `python examples/run_pipeline_mars.py --date 2025-09-29 --time 12 --step "1/to/2/by/1" --max-level 137` | Requires ECMWF MARS credentials (`.ecmwfapirc`) |

### 3️. Check the outputs

After running a pipeline, the processed GRIB2 files are written to `./out_grib2/`.

Each file corresponds to one forecast step (e.g. `dispf2025072912`).

Found a bug or have feedback? Please [open an issue](https://gitlab.phaidra.org/flexpart/flexprep/-/issues) or contact nina.burgdorfer@meteoswiss.ch