aitop 0.16.0
AI/ML system monitor with mixed-vendor GPU/NPU telemetry, AI-process intelligence, and Linux + experimental macOS support
License: MIT
Requires Python: >=3.9

# AITop - AI-Focused System Monitor

[![PyPI version](https://badge.fury.io/py/aitop.svg)](https://badge.fury.io/py/aitop)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Copyright © 2025 Alexander Warth. All rights reserved.

Current version: 0.16.0

Release highlights:
- Linux-first + experimental macOS coverage for mixed accelerator environments.
- Mixed-vendor GPU + NPU visibility in one workflow (including experimental AMD/Intel NPUs).
- Runtime header stays signal-focused without theme/release labels.

## Key Capabilities

- Mixed-accelerator observability in one TUI: NVIDIA/AMD/Intel GPUs plus experimental AMD/Intel NPUs and experimental Apple GPUs on macOS.
- Real-time accelerator monitoring: GPU/NPU utilization, VRAM, thermals, power, and driver/framework versions.
- AI workload intelligence: detects 50+ ML/AI frameworks with per-process metrics, job grouping, distributed topology, and straggler hints.
- Fast operator workflow: Overview, AI, GPU, and conditional NPU tabs with interactive filtering (`/`, `Ctrl+F`) and tree mode (`t`).
- Safe process control: htop-style signal actions with safeguards for critical and distributed AI jobs.
- Snapshot + trend visibility: overview sparklines and JSON snapshot export (`Shift+S`) for offline analysis.
- Platform coverage: Linux-first production workflow plus experimental macOS Apple GPU inventory support.
- Rare mixed-vendor + NPU coverage: one workflow across NVIDIA/AMD/Intel GPUs and experimental AMD/Intel NPUs.
- Low overhead by design: adaptive scheduling, caching, and resilient telemetry fallbacks.

AITop is a command-line system monitor built for AI/ML workloads across mixed accelerator environments.

> **Beta Phase Notice**: AITop is in advanced beta (v0.16.0) with active hardening, documentation updates, and expanded regression coverage.

## Features & Enhancements

![AITop Overview Mode](https://gitlab.com/CochainComplex/aitop/-/raw/main/screenshots/overview.png)
*AITop's main overview screen showing system-wide metrics with GPU, CPU, and memory status*

- **Real-time GPU Monitoring**
  - Utilization metrics (compute, memory)
  - Memory usage and allocation
  - Temperature and power consumption
  - Framework version detection (CUDA, ROCm, driver versions)
  - Cross-architecture CPU vendor/model extraction improvements for Linux (`/proc/cpuinfo` key variants and ARM implementer mapping)

- **AI Workload Focus**
  - Detection of AI/ML processes via pattern matching
  - Interactive filtering across AI, GPU, and conditional NPU tabs (`/` or `Ctrl+F`, Esc/Ctrl+L to clear) with field-aware queries (`name:`, `pid:`, `user:`, `cmd:`, `status:`, `gpu:`).
  - Tree-mode grouping of related AI processes (`t` toggles depth view) plus richer IO/NET metrics to help spot high-throughput jobs.
  - Job-topology inference groups related workers (`job_id`, rank/world-size hints) and flags potential stragglers in distributed workloads.
  - Overview process table now surfaces per-task device placement (`CPU`, `GPUx`, `NPUx`) with adaptive compacting (up to 3 devices on smaller systems, 2 on larger fleets, plus `+N` overflow).
  - AI Processes now includes a grouped `By Device` summary (process count + per-device utilization/memory footprint) for faster placement triage.
  - Recently sleeping AI processes are retained for a 120-second grace window to avoid transient disappearance during short telemetry gaps.

![AITop Process View](https://gitlab.com/CochainComplex/aitop/-/raw/main/screenshots/process_view.png)
*Detailed process monitoring with AI workload identification and resource usage tracking*
  - AI/ML framework pattern recognition (50+ frameworks)
  - Training job and distributed workload detection
  - Process classification and identification

- **Process Management**
  - htop-like process killing with signal selection
  - Comprehensive safety checks (permissions, system-critical processes)
  - AI-specific warnings for training jobs and distributed workloads
  - Support for SIGTERM, SIGKILL, SIGHUP, SIGINT, SIGQUIT, SIGSTOP, SIGCONT
  - Extra confirmation for dangerous signals (SIGKILL)
  - Automatic privileged fallback: prompts for sudo password when normal kill lacks permission.
  - Snapshot exports (`Shift+S`) writing JSON reports for offline analysis and troubleshooting.

- **Multi-Vendor Support**
  - NVIDIA GPUs (via nvidia-smi with robust query fallbacks)
  - AMD GPUs (via ROCm with rocm-smi preferred and amd-smi fallback)
  - Intel GPUs (via intel_gpu_top JSON stream parsing with timeout-safe partial reads)
  - Experimental macOS Apple GPUs (via `system_profiler` inventory; live metrics are currently limited)
  - Experimental AMD NPUs (via `xrt-smi` with accel sysfs fallback)
  - Experimental Intel NPUs (via accel sysfs `ivpu` telemetry counters)
  - Overview accelerator cards now label NPU devices as `NPU` (instead of `GPU`) when NPU vendors are detected.
  - GPU/NPU process trees now include explicit per-process device IDs (`GPU0`, `NPU0`, etc.) for clearer attribution.

![AITop GPU Panel](https://gitlab.com/CochainComplex/aitop/-/raw/main/screenshots/gpu_panel.png)
*Comprehensive GPU monitoring with detailed metrics for each detected GPU*

- **Customizability**
  - Configure displayed metrics
  - Customize refresh rates
  - Choose different output views (compact, detailed, graphical)

- **Enhanced Interactive UI**
  - Dynamic, color-coded displays with advanced color management
  - True color support for modern terminals with automatic detection
  - Intelligent color fallback for 256-color and basic terminals
  - Adaptive rendering for lower system impact
  - Improved process-specific monitoring with AI detection
  - Terminal-aware theme selection with manual override support

- **Performance Optimizations**
  - Efficient metric polling with minimal impact on GPU workloads
  - Low CPU and memory overhead with adaptive scheduling
  - Smart caching for optimal rendering performance
  - Multi-threaded data collection with prioritized updates
  - Error resilience with exponential backoff when needed
  - Memory-efficient data structures with proper cleanup
  - Compact sparkline history in the overview tab surfaces recent CPU/MEM/GPU trends for faster situational awareness.

## Experimental Features

- ARM CPU vendor/model inference on Linux via `/proc/cpuinfo` key variants and implementer mapping (best-effort; may show `Unknown`).
- Intel GPU monitoring via `intel_gpu_top` JSON stream sampling (driver/tool support varies; metrics may be partial).
- Experimental macOS Apple GPU monitoring via `system_profiler SPDisplaysDataType -xml` inventory with conservative cached polling (best-effort; runtime utilization/process attribution is limited).
- AMD/Intel NPU telemetry via vendor tools or accel sysfs (best-effort; counters and process attribution may be incomplete).
- AMD NPU monitoring via `xrt-smi` reports and accel sysfs counters (best-effort; metrics vary by driver/toolchain).
- Intel NPU monitoring via accel sysfs counters (`npu_busy_time_us`, `npu_memory_utilization`) with `ivpu` driver metadata when present.

## Installation

### Quick Install (Recommended)

Install AITop directly from PyPI:

```bash
pip install aitop
```

For development features, install with extra dependencies:

```bash
pip install aitop[dev]
```

### From Source

1. **Clone the Repository**

   ```bash
   git clone https://gitlab.com/CochainComplex/aitop.git
   cd aitop
   ```

2. **Select Python with pyenv**

   This project uses `pyenv` (not `venv`) for local Python selection.
   Python 3.9+ is supported.

   ```bash
   pyenv install 3.9.20
   pyenv local 3.9.20
   ```

3. **Install Dependencies**

   ```bash
   pyenv exec pip install -e ".[dev]"
   ```

   Dependency files are split by purpose:
   - `requirements.txt` (runtime)
   - `requirements-dev.txt` (lint/test tooling)
   - `requirements-docs.txt` (docs build)

## Development Tooling

Use `pyenv exec` to run tools so they pick up the project-local Python.

```bash
pyenv exec python -m ruff format --check aitop tests setup.py scripts
pyenv exec python -m ruff check aitop tests setup.py scripts
pyenv exec python scripts/check_complexity.py
pyenv exec python -m mypy --strict aitop
pyenv exec pytest --cov=aitop --cov-branch --cov-config=pyproject.toml --cov-fail-under=40 -q
pyenv exec python -m pip_audit -r requirements.txt -r requirements-dev.txt -r requirements-docs.txt
pyenv exec python scripts/check_architecture.py
pyenv exec pre-commit run --all-files
```

Operational command set and troubleshooting notes live in `RUNBOOK.md`.

### GPU Dependencies

No additional Python packages are required beyond `psutil` for GPU/NPU support.
NPU integration is handled via system drivers/tools and sysfs telemetry.

   - **NVIDIA GPUs**

     Ensure NVIDIA drivers are installed and nvidia-smi is accessible.
     Supports NVIDIA driver 400+ with compatible nvidia-smi versions.

   - **AMD GPUs**

     Install ROCm as per [ROCm Installation Guide](https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html).
     Supports ROCm 4.x through 7.x with 6-method fallback chain for version detection.
     Uses `rocm-smi` when available, with `amd-smi` fallback when `rocm-smi` is unavailable.

   - **Intel GPUs**

     Requires intel_gpu_top tool (part of intel-gpu-tools package).
     Uses non-interactive `intel_gpu_top -L`/`-J` probes with stream-safe parsing.
     Intel GPU support provides utilization + process visibility where exposed by intel-gpu-tools.

   - **Apple GPUs on macOS (Experimental)**

     Requires `system_profiler` (built into macOS) for GPU inventory.
     AITop uses `system_profiler SPDisplaysDataType -xml` and normalizes detected
     VRAM/unified-memory strings (for example `GB/MB/KB` variants) into internal MB units.
     Runtime utilization/process telemetry is currently limited on macOS without
     privileged tooling; AITop keeps this path conservative to avoid unstable probes.
     Optional live utilization can be enabled with:
     `AITOP_ENABLE_MACOS_POWERMETRICS=1` (requires `powermetrics`; root/admin may be required).
     When enabled, utilization is derived from `powermetrics --samplers gpu_power`
     active GPU residency percentages (best-effort, no per-process attribution).

   - **AMD NPUs (Experimental)**

     Requires AMD NPU-capable kernel driver support (`amdxdna`) and typically
     the XRT toolchain exposing `xrt-smi`.
     Prefers `xrt-smi` telemetry reports and falls back to accel sysfs when available.
     Coverage is best-effort and depends on `amdxdna`/XRT versions.

   - **Intel NPUs (Experimental)**

     Requires Intel VPU kernel support (`ivpu`) exposing `/sys/class/accel`.
     Uses accel sysfs telemetry (`npu_busy_time_us`, `npu_memory_utilization`,
     and frequency attributes when available).
     Coverage is best-effort and kernel-version dependent.

## Quick Start

Launch AITop with the following command:

```bash
# Start AITop with default settings
aitop

# Enable debug logging
aitop --debug

# Customize performance parameters
aitop --update-interval 0.8 --process-interval 3.0 --gpu-interval 1.5

# Select a specific theme
aitop --theme nord
```

## Usage

### Command Line Options

AITop now features several command-line options to customize behavior:

```bash
# Get help on all options
aitop --help

# Basic Options
--debug                Enable debug logging
--log-file FILE        Path to log file (default: aitop.log)

# Performance Options
--update-interval N    Base data update interval in seconds (default: 0.5)
--process-interval N   Process collection interval in seconds (default: 2.0)
--gpu-interval N       Full GPU info interval in seconds (default: 1.0)
--render-interval N    UI render interval in seconds (default: 0.2)
--workers N            Number of worker threads (default: 3)

# Display Options
--theme THEME          Override theme selection (e.g., monokai_pro, nord)
--list-themes          List available themes and exit
--no-adaptive-timing   Disable adaptive timing based on system load
```

### Interactive Controls

- `?`: Show the full help overlay with key mappings and filter examples.
- `Shift+S`: Export the current snapshot (JSON) into the working directory.
- `/` or `Ctrl+F`: Enter the process filter prompt (`name:`, `pid:`, `user:`, `cmd:`, `status:`, `gpu:` terms supported), Esc/Ctrl+L clears the filter.
- `t`: Toggle tree-structured grouping for the AI Processes tab (useful for drilling into branches).

### Privileged Kill Mode

When a process signal requires elevated privileges, AITop can retry with sudo:

```bash
aitop
```

AITop first attempts a normal kill. If permission is denied, it asks for sudo password
in the TUI (hidden input) and uses it only for that single privileged kill action.

### Theme Configuration

AITop includes an enhanced theme system that automatically adapts to your terminal environment:

- **Intelligent Detection**: Automatically selects the optimal theme based on:
  - Terminal capability hints (`TERM`, `COLORTERM`)
  - Color support capabilities (true color, 256 colors, or basic)
  - Curses palette-mutation support (`can_change_color()`) with safe fallbacks

- **Manual Override**: Set preferred theme using environment variable:
  ```bash
  # Enable 256-color support (required for some terminals)
  export TERM=xterm-256color

  # Set theme before running aitop
  export AITOP_THEME=default
  aitop
  ```

- **Available Themes**:
  - `default`: Standard theme based on htop colors
  - `graphite_modern`: Neutral graphite palette (default)
  - `monokai_pro`: Modern dark theme with vibrant, carefully balanced colors
  - `nord`: Arctic-inspired color palette optimized for eye comfort
  - `solarized_dark`: Scientifically designed for optimal readability
  - `material_ocean`: Modern theme based on Material Design principles
  - `stealth_steel`: Sleek gray-based palette with subtle color accents
  - `forest_sanctuary`: Nature-inspired palette with rich greens and earthen tones
  - `cyberpunk_neon`: Futuristic neon color scheme with vibrant accents

Each theme is carefully crafted for specific use cases:
- `monokai_pro`: Features a vibrant yet balanced color scheme with distinctive progress bars (▰▱)
- `nord`: Offers a cool, arctic-inspired palette that reduces eye strain with elegant progress bars (━─)
- `solarized_dark`: Uses scientifically optimized colors for maximum readability with classic block indicators (■□)
- `material_ocean`: Implements Material Design principles with circular progress indicators (●○)
- `stealth_steel`: Provides a professional, minimalist look with half-block indicators (▀░)
- `forest_sanctuary`: Delivers a natural, calming experience with bold filled blocks (`▮`) and blank unfilled segments for clearer contrast
- `graphite_modern`: Default theme with neutral graphite tones and the same high-contrast filled-block (`▮`) plus blank-unfilled bar style
- `cyberpunk_neon`: Features a high-contrast neon palette with classic block indicators (█░)

### Color Support

AITop now features advanced color management:
- Automatic detection of terminal color capabilities
- True color support (16 million colors) for modern terminals
- Intelligent fallback for terminals with limited color support
- Color caching for optimal performance
- Smooth color approximation when exact colors aren't available

### Performance Features

AITop includes several advanced performance features:

- **Adaptive Timing**: Automatically adjusts update frequency based on system load
- **Staggered Collection**: Different metrics are collected at optimized intervals:
  - Fast metrics (CPU, memory usage) are updated more frequently
  - Expensive metrics (full GPU details, process scanning) are updated less frequently
- **Smart Caching**: Cache system with TTL (Time-To-Live) for efficient data retrieval
- **Error Resilience**: Exponential backoff on errors to prevent resource exhaustion
- **Optimized Rendering**: Differential screen updates to minimize CPU usage

### Debug Mode

AITop supports an enhanced debug mode that can be enabled with the `--debug` flag. When enabled:
- Creates a detailed log file (default: `aitop.log` in the current directory)
- Logs comprehensive debug information including:
  - Application initialization and shutdown sequences
  - Data collection events and timing
  - UI rendering updates and performance metrics
  - Detailed error traces with context
  - System state changes and GPU detection
  - Theme detection and color management
  - Collection statistics for performance analysis
- Useful for troubleshooting issues or monitoring performance


AITop provides an interactive interface with the following controls:

- **Navigation**
  - **Left/Right Arrow Keys**: Switch between tabs
  - **Up/Down Arrow Keys**: Navigate and select processes in AI Processes, GPU, and NPU tabs (when NPU is detected)

- **Process Management** (AI Processes, GPU, and NPU tabs)
  - **'k'**: Kill selected process (opens signal selection menu)
  - Use **Up/Down** to select a process, then press **'k'** to send a signal
  - Choose from SIGTERM (graceful), SIGKILL (force), or other signals
  - Safety checks prevent accidental system damage
  - Works in AI Processes tab (full process list), GPU tab (GPU-using processes), and NPU tab (NPU-using processes when detected)

- **Process Sorting**
  - **'c'**: Sort by CPU usage
  - **'m'**: Sort by memory usage
  - **'h'**: Toggle sort order (ascending/descending)

- **General**
  - **'q'**: Quit application
  - **'r'**: Force refresh display

### Interface Tabs

- **Overview**: System-wide metrics including CPU, memory, overall GPU usage, and an AI Jobs summary (workers + stragglers per inferred job).
- **AI Processes**: Lists detected AI/ML processes with detailed metrics.
- **GPU**: Detailed GPU metrics per vendor, including utilization, temperature, and power consumption.
- **NPU** (conditional): Appears only when AMD/Intel NPU devices are detected; shows accelerator telemetry and process activity using the same interaction model as the GPU tab.
- **Memory**: System memory statistics and usage.
- **CPU**: CPU usage and performance statistics.

For detailed project structure and component documentation, see [STRUCTURE.md](https://gitlab.com/CochainComplex/aitop/-/blob/main/STRUCTURE.md).

## Troubleshooting

### Color Rendering Issues

**Problem**: Colors look strange, washed out, or theme changes don't work properly (especially on Pop OS Cosmic DE, Alacritty, or similar modern terminals).

**Root Cause**: Modern terminals support true color (`COLORTERM=truecolor`) but don't support curses palette modification (`can_change_color()` returns False). This causes hex-based themes to fall back to 256-color approximation, which may not match the intended colors.

**Solutions**:

1. **Use a 256-color theme** (Recommended):
   ```bash
   export AITOP_THEME=solarized_dark
   aitop
   ```
   Or:
   ```bash
   aitop --theme solarized_dark
   ```

   Available 256-color themes: `solarized_dark`, `material_ocean`

2. **Use the default theme** (8-color, most compatible):
   ```bash
   export AITOP_THEME=default
   aitop
   ```

3. **Force 256-color detection** (prevents hex theme selection):
   ```bash
   unset COLORTERM
   aitop
   ```

4. **List all available themes**:
   ```bash
   aitop --list-themes
   ```

**Debug color detection**:
```bash
aitop --debug
# Check aitop.log for terminal capability detection:
# - TERM and COLORTERM values
# - max_colors detected
# - can_change_color value
# - Selected theme
```

### Terminal Size Issues

**Problem**: AITop displays incorrectly or crashes on startup.

**Solution**: Ensure your terminal is at least 80x24 characters. Resize the terminal window or adjust font size.

### GPU Not Detected

**Problem**: GPUs are not shown in the GPU tab.

**Solutions**:

1. **NVIDIA**: Ensure `nvidia-smi` is accessible:
   ```bash
   nvidia-smi
   ```

2. **AMD**: Ensure `rocm-smi` is installed and accessible (fallback: `amd-smi`):
   ```bash
   rocm-smi
   ```

3. **Intel**: Ensure `intel_gpu_top` is installed:
   ```bash
   intel_gpu_top -l
   ```

### Performance Issues

**Problem**: AITop uses too much CPU or updates too slowly.

**Solutions**:

1. **Adjust update intervals**:
   ```bash
   aitop --update-interval 1.0 --process-interval 3.0 --render-interval 0.3
   ```

2. **Reduce worker threads**:
   ```bash
   aitop --workers 2
   ```

3. **Disable adaptive timing** (if causing issues):
   ```bash
   aitop --no-adaptive-timing
   ```

### Theme Not Changing

**Problem**: Setting `--theme` or `AITOP_THEME` doesn't change colors.

**Diagnostic**:
```bash
aitop --list-themes  # Verify theme name is correct
aitop --theme invalid_name  # Should show error and suggestions
```

**Solution**:
- Theme names are case-sensitive
- Use exact names from `--list-themes` output
- Try clearing terminal and restarting AITop
- Check `aitop.log` with `--debug` for theme loading errors

## Development

Run the same quality gate locally as CI (Python 3.9 baseline):

```bash
pyenv install 3.9.20
pyenv local 3.9.20
pyenv exec pip install -e ".[dev]"
pyenv exec pre-commit install
pyenv exec python -m ruff format --check aitop tests setup.py scripts
pyenv exec python -m ruff check aitop tests setup.py scripts
pyenv exec python scripts/check_complexity.py
pyenv exec python -m mypy --strict aitop
pyenv exec pytest --cov=aitop --cov-branch --cov-config=pyproject.toml --cov-fail-under=40 -q
pyenv exec python -m pip_audit -r requirements.txt -r requirements-dev.txt -r requirements-docs.txt
pyenv exec python scripts/check_architecture.py
```

For failure triage and remediation commands, see `RUNBOOK.md`.

## Requirements

- **Python 3.9+**
- **NVIDIA Drivers** (for NVIDIA GPU support)
- **ROCm** (for AMD GPU support)
- **intel-gpu-tools (`intel_gpu_top`)** (for Intel GPU support)

## Contributing

Contributions are welcome! Please follow these steps to contribute:

1. **Fork the Repository**
2. **Create Your Feature Branch**

   ```bash
   git checkout -b feature/AmazingFeature
   ```

3. **Commit Your Changes**

   ```bash
   git commit -m 'Add some AmazingFeature'
   ```

4. **Push to the Branch**

   ```bash
   git push origin feature/AmazingFeature
   ```

5. **Open a Pull Request**

   Discuss your changes and get feedback before merging.

## License

This project is licensed under the MIT License - see the [LICENSE](https://gitlab.com/CochainComplex/aitop/-/blob/main/LICENSE) file for details.

## Author

Alexander Warth
[Professional Website](https://warth.ai)

## Legal Disclaimer

AITop is an independent project and is not affiliated with, endorsed by, or sponsored by NVIDIA Corporation, Advanced Micro Devices, Inc. (AMD), or Intel Corporation. All product names, logos, brands, trademarks, and registered trademarks mentioned in this project are the property of their respective owners.

- NVIDIA®, CUDA®, and NVML™ are trademarks and/or registered trademarks of NVIDIA Corporation.
- AMD® and ROCm™ are trademarks and/or registered trademarks of Advanced Micro Devices, Inc.
- Intel® is a trademark and/or registered trademark of Intel Corporation.

The use of these trademarks is for identification purposes only and does not imply any endorsement by the trademark holders. AITop provides monitoring capabilities for GPU hardware but makes no guarantees about the accuracy, reliability, or completeness of the information provided. Use at your own risk.

## Acknowledgments

Special thanks to:
- The open-source community
- All contributors and users of AITop