mocea 2.0.6
Idle droplet monitor and auto-terminator.
Requires Python: >=3.12

# mocea

Idle droplet monitor and auto-terminator.

**mocea** (Monitoring and Management of digital OCEA droplets) is a lightweight agent that runs on a DigitalOcean droplet, detects when it's idle, and auto-terminates it via the DigitalOcean API (snapshot + destroy) to save costs. It runs as a systemd service and uses `psutil` for rich local metric collection.

## Installation

```bash
pip install mocea
```

## Quick Start

```bash
# Test the checks
mocea check

# Start in foreground with dry-run
mocea run --dry-run --idle-minutes 5

# Install as systemd service
sudo mocea install
```

## CLI Commands

```
mocea run          Start the monitoring agent
mocea check        Run all checks once and display results
mocea status       Show service status and current check results
mocea config       Show active configuration
mocea cloudinit    Generate cloud-init user-data for droplet bootstrap
mocea install      Install mocea as a systemd service
mocea uninstall    Remove mocea systemd service
```

### `mocea run`

```
Options:
  -c, --config PATH       Config file path
  --idle-minutes INTEGER  Override idle timeout (minutes)
  --dry-run               Log actions but don't execute
  --log-level [DEBUG|INFO|WARNING]
```

## Configuration

Config file: `/etc/mocea/config.toml` (or `~/.config/mocea/config.toml`)

```toml
idle_minutes = 30
check_interval = 60
min_uptime_minutes = 10

[checks.cpu]
enabled = true
threshold = 5.0          # percent, below = idle

[checks.process]
enabled = true
names = ["python", "ffmpeg", "jupyter"]

[checks.ssh]
enabled = true

[checks.load]
enabled = false
threshold = 0.3

[checks.network]
enabled = false
threshold_kbps = 10
# interface = "eth0"      # omit for all non-loopback interfaces

[checks.gpu]
enabled = false

[checks.heartbeat]
enabled = false
file = "/tmp/mocea-heartbeat"
stale_minutes = 15

[action]
type = "api"              # "api" | "shutdown"

# Optional: api action settings
# [action.api]
# snapshot_before_destroy = true

[logging]
level = "INFO"
# file = "/var/log/mocea.log"  # omit for stdout only
```

Priority: CLI flags > config file > defaults.

## Checks

| Check | Signal | Idle when |
|-------|--------|-----------|
| **cpu** | `psutil.cpu_percent()` | Below threshold (default 5%) |
| **process** | `psutil.process_iter()` | No configured process names running |
| **ssh** | `psutil.net_connections()` port 22 | No ESTABLISHED SSH sessions |
| **load** | `psutil.getloadavg()` | 1-min load below threshold (default 0.3) |
| **network** | `psutil.net_io_counters()` | Throughput below threshold (default 10 KB/s) |
| **gpu** | `nvidia-smi` | GPU utilization below threshold (default 5%) |
| **heartbeat** | File mtime | File missing or stale (default 15 min) |

All enabled checks must report idle (AND logic) for the entire `idle_minutes` duration before the action triggers.

## Actions

| Action | Description |
|--------|-------------|
| **api** | Direct DO API: snapshot then destroy (default, requires `DO_API_TOKEN`) |
| **shutdown** | Simple `shutdown -h now` (no snapshot, for disposable droplets) |

## Safety

- **Minimum uptime**: Won't terminate within first 10 minutes after boot
- **Dry-run mode**: `--dry-run` logs what would happen without executing
- **Lock file**: Prevents multiple instances
- **Fail-safe**: If a check errors, it's treated as "active" (keeps running)
- **Heartbeat file**: Any process can touch `/tmp/mocea-heartbeat` to prevent termination

## Managing the Systemd Service

Once installed with `sudo mocea install`, mocea runs as a standard systemd service called `mocea.service`. Here's how to manage it.

### Checking Status

```bash
# mocea's built-in status (service + check results)
sudo mocea status

# Or use systemctl directly
sudo systemctl status mocea.service
```

### Viewing Logs

mocea logs to the systemd journal. Use `journalctl` to read them:

```bash
# Recent logs
sudo journalctl -u mocea.service -n 50

# Follow logs in real time (like tail -f)
sudo journalctl -u mocea.service -f

# Logs since last boot
sudo journalctl -u mocea.service -b

# Logs from the last hour
sudo journalctl -u mocea.service --since "1 hour ago"
```

### Stopping and Starting

```bash
# Stop mocea (it will restart on next boot since it's still enabled)
sudo systemctl stop mocea.service

# Start it again
sudo systemctl start mocea.service

# Restart (stop + start)
sudo systemctl restart mocea.service
```

### Disabling and Re-enabling

"Enabled" means mocea starts automatically on boot. "Disabled" means it won't.

```bash
# Stop AND prevent auto-start on boot
sudo systemctl disable --now mocea.service

# Re-enable auto-start (and start it now)
sudo systemctl enable --now mocea.service
```

### Updating mocea

mocea is installed in a virtualenv at `/opt/mocea`. To upgrade, use the venv's pip — **not** the system pip (which will fail with `externally-managed-environment` on modern Debian/Ubuntu):

```bash
# Stop the service
sudo systemctl stop mocea.service

# Upgrade using the venv's pip
sudo /opt/mocea/bin/pip install --upgrade --no-cache-dir mocea

# Restart the service to pick up the new code
sudo systemctl start mocea.service

# Verify the new version
mocea --version
```

If the mocea binary path changed (e.g. you recreated the virtualenv), reinstall the service:

```bash
sudo mocea uninstall
sudo /path/to/new/venv/bin/mocea install
```

### Uninstalling

```bash
sudo mocea uninstall
```

This stops the service, disables it, and removes the unit file.

### Troubleshooting

| Symptom | Command | What to look for |
|---------|---------|-----------------|
| Service won't start | `sudo journalctl -u mocea.service -n 30` | Python errors, missing config |
| Exit code 203/EXEC | `cat /etc/systemd/system/mocea.service` | `ExecStart` must be an absolute path; reinstall with `sudo mocea install` |
| Service keeps restarting | `sudo systemctl status mocea.service` | Shows restart count and exit code |
| Not sure if running | `sudo systemctl is-active mocea.service` | Prints `active` or `inactive` |

The service is configured to restart automatically on failure after a 30-second delay. If it keeps crashing, check the logs with `journalctl`.

## Cloud-Init Bootstrap

Generate cloud-init user-data to auto-install mocea on new droplets:

```bash
mocea cloudinit > user-data.yaml
```

## Development

```bash
git clone https://github.com/ksteptoe/mocea.git
cd mocea
make bootstrap    # Create venv and install dependencies
make test         # Run tests
make lint         # Run linter
make format       # Format code
```