access-mopper 2.3.0a24
ACCESS Model Output Post-Processor, maps raw model output to CMIP-style defined variables and produce post-processed output using CMOR3
License: Apache-2.0
Requires Python: >=3.11
Keywords: ACCESS, post-processing

# ACCESS Model Output Post-Processor (ACCESS-MOPPeR) v2

[![Documentation Status](https://readthedocs.org/projects/access-mopper-v2/badge/?version=latest)](https://access-mopper-v2.readthedocs.io/en/latest/?badge=latest)
[![PyPI version](https://badge.fury.io/py/access_mopper.svg)](https://badge.fury.io/py/access_mopper)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/access_mopper.svg)](https://anaconda.org/conda-forge/access_mopper)

ACCESS-MOPPeR is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets. This version represents a significant rewrite focusing on usability, flexibility, and integration with modern Python workflows.

## Key Features

- **Python API** for integration into notebooks and scripts
- **Batch processing system** for HPC environments with PBS
- **Real-time monitoring** with web-based dashboard
- **Flexible CMORisation** of individual variables
- **Dask-enabled** for scalable parallel processing
- **Cross-platform compatibility** (not limited to NCI Gadi)
- **CMIP6 and CMIP7 FastTrack support**

## Installation

ACCESS-MOPPeR requires Python >= 3.11. Install with:

```bash
pip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit
pip install .
```

## Quick Start

### Interactive Usage (Python API)

```python
import glob
from access_mopper import ACCESS_ESM_CMORiser

# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")

# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
    input_paths=files,
    compound_name="Amon.pr",  # table.variable format
    experiment_id="historical",
    source_id="ACCESS-ESM1-5",
    variant_label="r1i1p1f1",
    grid_label="gn",
    activity_id="CMIP",
    output_path="/path/to/output"
)

# Run CMORisation
cmoriser.run()
cmoriser.write()
```

### Batch Processing (HPC/PBS)

For large-scale processing on HPC systems:

1. **Create a configuration file** (`batch_config.yml`):

```yaml
variables:
  - Amon.pr
  - Omon.tos
  - Amon.ts

experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn

input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"

file_patterns:
  Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
  Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
  Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"

# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"

worker_init: |
  module load conda
  conda activate your_environment
```

2. **Submit batch job**:

```bash
mopper-cmorise batch_config.yml
```

3. **Monitor progress** at http://localhost:8501

## Batch Processing Features

The batch processing system provides:

- **Parallel execution**: Each variable processed as a separate PBS job
- **Real-time monitoring**: Web dashboard showing job status and progress
- **Automatic tracking**: SQLite database maintains job history and status
- **Error handling**: Failed jobs can be easily identified and resubmitted
- **Resource optimization**: Configurable CPU, memory, and storage requirements
- **Environment management**: Automatic setup of conda/module environments

### Monitoring Tools

- **Streamlit Dashboard**: Real-time web interface at http://localhost:8501
- **Command line**: Use standard PBS commands (`qstat`, `qdel`)
- **Database**: SQLite tracking at `{output_folder}/cmor_tasks.db`
- **Log files**: Individual stdout/stderr for each job

### File Organization

```
work_directory/
├── batch_config.yml          # Your configuration
├── cmor_job_scripts/          # Generated PBS scripts and logs
│   ├── cmor_Amon_pr.sh       # PBS script
│   ├── cmor_Amon_pr.py       # Python processing script
│   ├── cmor_Amon_pr.out      # Job output
│   └── cmor_Amon_pr.err      # Job errors
└── output_folder/
    ├── cmor_tasks.db         # Progress tracking
    └── [CMORised files]      # Final output
```

## Documentation

- **Getting Started**: `docs/source/getting_started.rst`
- **Example Configuration**: `src/access_mopper/examples/batch_config.yml`
- **API Reference**: [Coming soon]

## Current Limitations

- **Alpha version**: Intended for evaluation only
- **Ocean variables**: Limited support in current release
- **Variable mapping**: Under review for CMIP6/CMIP7 compliance

## Support

- **Issues**: Submit via GitHub Issues
- **Questions**: Contact ACCESS-NRI support
- **Contributions**: Welcome via Pull Requests

## License

ACCESS-MOPPeR is licensed under the Apache-2.0 License.

---

**Background**: ACCESS-MOPPeR v2 is a complete rewrite using modern Python libraries (xarray, dask) instead of CMOR, providing improved flexibility and integration with contemporary data science workflows.