q2netcdf 0.4.0
Manipulate Rockland Scientific's Q-files
Requires Python: >=3.10
Keywords: ISDP, MicroRider, Q-file, Rockland

# q2netcdf

[![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/mousebrains/q2netcdf/actions/workflows/ci.yml/badge.svg)](https://github.com/mousebrains/q2netcdf/actions/workflows/ci.yml)
[![Coverage](https://codecov.io/gh/mousebrains/q2netcdf/branch/main/graph/badge.svg)](https://codecov.io/gh/mousebrains/q2netcdf)

> **Note**: The `mergeqfiles` tool is compatible with Python 3.7+ for deployment on legacy systems (e.g., MicroRider instruments), though only Python 3.10+ is actively tested in CI.

Python tools for manipulating [Rockland Scientific's](https://rocklandscientific.com) Q-files as generated by the [ISDP data logger](https://rocklandscientific.com/news/rockland-data-logger/).

## Features

- **Q-file to NetCDF Conversion**: Convert binary Q-files to CF-1.8 compliant NetCDF format
- **Header Parsing**: Extract metadata, sensor configurations, and timing information
- **Data Extraction**: Read scalar channels and frequency spectra from binary records
- **File Manipulation**: Merge, reduce, and inspect Q-files
- **Configuration Generation**: Create syntactically correct `isdp.cfg` files
- **Format Support**: Handles Q-file versions 1.2 and 1.3
- **Sensor Mapping**: Built-in support for 200+ sensor identifier codes

## Command-Line Tools

- **`q2netcdf`** - Translate Q-files into CF-1.8 compliant NetCDF files
- **`QFile`** - Dump header and data records from Q-files
- **`QHeader`** - Display header record information
- **`QReduce`** - Reduce Q-file size by removing records
- **`mergeqfiles`** - Merge multiple Q-files (designed for MicroRider/Slocum Glider integration)
- **`mkISDPcfg`** - Generate syntactically correct `isdp.cfg` configuration files

## Quick Start

```bash
# Display Q-file header information
QHeader data.q

# Convert Q-file to NetCDF
q2netcdf data.q -o output.nc

# Dump all records from Q-file
QFile data.q --logLevel INFO
```

## Installation

### Using pipx (Recommended)

The command-line scripts in this package may be installed using [pipx](https://pipx.pypa.io/stable/installation/):

```bash
python3 -m pipx install git+https://github.com/mousebrains/q2netcdf
```

To update to the latest version:

```bash
python3 -m pipx install --force git+https://github.com/mousebrains/q2netcdf
```

### From Source

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e .
```

### For Development

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest  # Run tests
```

## Installation on the MicroRider

For deployment on Rockland Scientific MicroRider instruments:

1. Transfer [src/q2netcdf/microrider.py](microrider.py) [src/q2netcdf/microrider.cfg](microrider.cfg) to the MicroRider's data directory.
2. Dismount the data directory on your host computer.
3. Connect to the MicroRider via a serial connection using your favorite terminal emulator at 115,200 baud.
4. `ls data` to verify the files are present. This will cause the MicroRider to unmount data from being read-only to the MicroRider and then remount it as read-write. This will typically cause the serial connection to drop then be reestablished. You might have to restart your terminal emulator's connection.
5. `mv data/microrider.py /usr/local/bin/mergeqfiles` to install my version of microrider.py as mergeqfiles.
6. Test with `mergeqfiles --help` to make sure the installation was successful.
7. You can disconnect the MicroRider's serial connection.
8. You should now run a simulation on the Slocum glider to verify Q-files are being uploaded to the science computer's storage as .mri files.
9. Diagnostic information is written into the data/mergeqfiles.log file on the MicroRider.
10. You can modify data/mergeqfiles.cfg to adjust the information included in the .mri file sent to the glider.

This is designed for integration with [TWR's Slocum Glider uRider proglet](https://www.teledynemarine.com/brands/webb-research/slocum-glider).

## Usage Examples

### Converting Q-files to NetCDF

```bash
q2netcdf data.q -o output.nc
```

### Inspecting Q-file Headers

```bash
# Show full header with channel and spectra names
QHeader data.q

# Show condensed header
QHeader data.q --nothing

# Show only configuration
QHeader data.q --channels --spectra --frequencies
```

### Examining Q-file Data Records

```bash
# Dump all records
QFile data.q

# Limit number of records displayed
QFile data.q --maxRecords 10
```

### Merging Q-files

```bash
mergeqfiles -o merged.q file1.q file2.q file3.q
```

### Generating ISDP Configuration Files

```bash
mkISDPcfg --help
```

## File Format

Q-files are binary files containing oceanographic data from Rockland Scientific instruments. The format is documented in Rockland Technical Note TN-054.

### Q-file Structure
- **Header Record** (0x1729): File version, timestamp, channel/spectra identifiers, frequencies, configuration
- **Data Records** (0x1657): Timestamped measurements with scalar channels and frequency spectra

### Supported Versions
- **v1.2**: Original format with full metadata in each data record
- **v1.3**: Optimized format with reduced redundancy

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](documents/CONTRIBUTING.md) for guidelines.

### Development Setup

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest                    # Run tests
ruff check src/ tests/    # Lint
mypy src/                 # Type check
```

To run a module for testing `python3 -m src.q2netcdf.QHeader --help`

## Additional Documentation

- [CHANGELOG.md](documents/CHANGELOG.md) - Version history and changes
- [CONTRIBUTING.md](documents/CONTRIBUTING.md) - Contributor guidelines and development setup
- [SECURITY.md](documents/SECURITY.md) - Security policy and vulnerability reporting
- [IMPROVEMENTS_APPLIED.md](documents/IMPROVEMENTS_APPLIED.md) - Recent improvements and updates
- [CLAUDE.md](CLAUDE.md) - AI assistant guidance for working with this codebase
- [Development Documentation](docs/development/) - Internal development notes and analysis

## License

See LICENSE file for details.

## Author

**Pat Welch**
Email: pat@mousebrains.com

## Acknowledgments

Developed for use with [Rockland Scientific](https://rocklandscientific.com) oceanographic instruments and the ISDP data logger.