pkg-ext 0.3.5
Python package public API management, versioning, and changelog generation
Requires Python: >=3.13
Keywords: api, changelog, cli, package-management, versioning

# Pkg Ext

[![PyPI](https://img.shields.io/pypi/v/pkg-ext)](https://pypi.org/project/pkg-ext/)
[![GitHub](https://img.shields.io/github/license/EspenAlbert/pkg-ext)](https://github.com/EspenAlbert/pkg-ext)
[![codecov](https://codecov.io/gh/EspenAlbert/pkg-ext/graph/badge.svg?token=47B15SDYMF)](https://codecov.io/gh/EspenAlbert/pkg-ext)
[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://espenalbert.github.io/pkg-ext/)


A CLI tool for managing Python package public API, versioning, and changelog generation.

## Overview

`pkg-ext` tracks which symbols (functions, classes, exceptions) in your package are "exposed" (public) vs "hidden" (internal). It:
- Generates `__init__.py` with imports and `__all__` based on decisions stored in changelog entries
- Creates group modules (e.g., `my_group.py`) that re-export related symbols
- Maintains a structured changelog directory (`.changelog/`) per PR
- Bumps version based on changelog action types (make_public=minor, fix=patch, delete/rename=major)
- Writes a human-readable `CHANGELOG.md`
- Provides [stability decorators](docs/stability.md) (`@experimental`, `@deprecated`) with suppressible warnings
- Generates `_warnings.py` in target packages to avoid runtime pkg-ext dependency

## Installation

```bash
uv pip install pkg-ext
# or
pip install pkg-ext
```

## Core Concepts

### Symbol Reference IDs

Symbols are identified by `{module_path}.{symbol_name}`, e.g., `my_pkg.utils.parse_config`.

### Changelog Actions

Stored in `.changelog/{pr_number}.yaml` files using Pydantic discriminated unions:

| Action Type | Description | Version Bump | Key Fields |
|-------------|-------------|--------------|------------|
| `make_public` | Make symbol public | Minor | `group`, `details` |
| `keep_private` | Keep symbol internal | None | `full_path` |
| `fix` | Bug fix from git commit | Patch | `short_sha`, `message`, `changelog_message`, `ignored` |
| `delete` | Remove from public API | Major | `group` |
| `rename` | Rename with old alias | Major | `group`, `old_name` |
| `breaking_change` | Breaking API change | Major | `group`, `details` |
| `additional_change` | Non-breaking change | Patch | `group`, `details` |
| `group_module` | Assign module to a group | None | `module_path` |
| `release` | Version release marker | None | `old_version` |
| `experimental` | Mark as experimental | Patch | `target`, `group`/`parent` |
| `ga` | Graduate to GA | Patch | `target`, `group`/`parent` |
| `deprecated` | Mark as deprecated | Patch | `target`, `group`/`parent`, `replacement` |
| `max_bump_type` | Cap version bump | None | `max_bump`, `reason` |
| `chore` | Internal changes | Patch | `description` |

All actions inherit common fields: `name`, `ts`, `author`, `pr`.

The `breaking_change` and `additional_change` actions support optional fields for API diff:
- `change_kind: str | None` - machine-readable change type (e.g., `param_removed`, `default_changed`)
- `auto_generated: bool` - `true` when created by API diff, `false` for interactive actions
- `field_name: str | None` - field name for field-level changes

### Stability Targets

Stability actions (`experimental`, `ga`, `deprecated`) support three target levels:

| Target | Description | Required Field |
|--------|-------------|----------------|
| `group` | Entire group | `name` = group name |
| `symbol` | Single symbol | `group` + `name` = symbol name |
| `arg` | Function argument | `parent` = `{group}.{symbol}`, `name` = arg name |

### Public Groups

Groups organize related symbols. Configured in `.groups.yaml`:

```yaml
groups:
  - name: __ROOT__  # Top-level exports in __init__.py
    owned_refs: []
    owned_modules: []
  - name: my_group
    owned_refs:
      - my_pkg.utils.parse_config
    owned_modules:
      - my_pkg.utils
```

When a new symbol is exposed, the tool prompts you to select which group it belongs to. All symbols from the same module go to the same group.

## CLI Commands

```bash
pkg-ext [OPTIONS] COMMAND
```

### Global Options

| Option | Description |
|--------|-------------|
| `-p, --path, --pkg-path` | Package directory path (auto-detected if not provided) |
| `--repo-root` | Repository root (auto-detected from `.git`) |
| `--is-bot` | CI mode: no prompts, fail on missing decisions |
| `--skip-open` | Skip opening files in editor |
| `--tag-prefix` | Git tag prefix (e.g., `v` for `v1.0.0`) |

### Command Reference

| Category | Commands | Description | Docs |
|----------|----------|-------------|------|
| Workflow | `pre-change`, `pre-commit`, `post-merge` | Development lifecycle commands | [docs/workflows](docs/workflows/index.md) |
| Changelog | `chore`, `promote`, `release-notes` | Changelog management | [docs/changelog](docs/changelog/index.md) |
| Stability | `exp`, `ga`, `dep` | Stability level management | [docs/stability](docs/stability/index.md) |
| API | `diff-api`, `dump-api` | API comparison and export | [docs/api_commands](docs/api_commands/index.md) |
| Generation | `gen-docs`, `gen-examples`, `gen-tests` | Generate documentation and scaffolds | [docs/generate](docs/generate/index.md) |

### When to Use Workflow Commands

| Scenario | Command |
|----------|---------|
| Added or removed symbols | `pre-change` |
| Final validation before commit | `pre-commit` |
| Single command for everything | `pre-change --full` |
| CI/CD pipeline | `pre-commit` (bot mode) |

- **`pre-change`** handles interactive decisions (expose/hide symbols, delete/rename). Fast because it only generates example and test scaffolds.
- **`pre-commit`** validates all decisions are made (fails in bot mode if prompts needed), syncs generated files, regenerates docs, and runs the dirty check.
- **`pre-change --full`** combines both: runs interactive prompts, generates examples/tests, then syncs files and regenerates docs. The dirty check is skipped since you're still developing.

## Configuration

### User Config (`~/.config/pkg-ext/config.toml`)

```toml
[user]
editor = "cursor"  # or "code", "vim", etc.
skip_open_in_editor = false
```

### Project Config (`pyproject.toml`)

```toml
[tool.pkg-ext]
tag_prefix = "v"
file_header = "# Generated by pkg-ext"
commit_fix_prefixes = ["fix:", "fix(", "bugfix:", "hotfix:"]
commit_diff_suffixes = [".py", ".pyi"]
changelog_cleanup_count = 30  # Archive when count exceeds this
changelog_keep_count = 10     # Keep this many after cleanup
format_command = ["ruff", "format"]  # ruff check --fix always runs first
max_bump_type = "minor"  # Cap version bumps (patch, minor, major)
# after_file_write_hooks = ["extra-cmd {pkg_path}"]  # Custom post-write hooks
```

### Group Configuration

Define groups with explicit settings in `pyproject.toml`:

```toml
[tool.pkg-ext.groups.my_group]
dependencies = ["__ROOT__"]  # Groups this depends on
docs_exclude = ["internal_helper"]
docstring = "Utilities for common operations"
```

**Note:** Stability is not configured here. Use `pkg-ext exp/ga/dep` CLI commands to manage stability via changelog actions.

### Version Bump Limits

For pre-1.0.0 packages where breaking changes are expected, cap the version bump:

**Project-level** (applies to all PRs):

```toml
# pyproject.toml
[tool.pkg-ext]
max_bump_type = "minor"  # All PRs capped to minor
```

**Per-PR override** (`MaxBumpTypeAction` in changelog overrides config):

```yaml
# .changelog/{pr}.yaml
name: version_cap
type: max_bump_type
max_bump: patch
reason: Documentation-only release
ts: '2026-01-17T14:35:00+00:00'
```

### Dev Mode

The `pre-commit` command enables dev mode, which writes to `-dev` suffixed files:
- `.groups-dev.yaml` instead of `.groups.yaml`
- `CHANGELOG-dev.md` instead of `CHANGELOG.md`

This allows iterating on changelog entries during development without modifying the production files. The real files are only updated by `post-merge` after PR is merged.

## Generated Files

### Files Updated During PR

| File | Purpose | Editable |
|------|---------|----------|
| `.changelog/{pr}.yaml` | Changelog actions for this PR | Yes |
| `.groups-dev.yaml` | Group assignments (dev copy) | No |
| `CHANGELOG-dev.md` | Human-readable changelog (dev copy) | No |
| `{pkg}.api-dev.yaml` | API dump for dev comparison (gitignored) | No |
| `{pkg}/__init__.py` | Package exports (VERSION unchanged) | No |
| `{pkg}/{group}.py` | Group re-export modules | No |
| `{pkg}/_warnings.py` | Stability warning decorators | No |
| `docs/**/*.md` | API documentation | Yes (outside markers) |
| `{group}_examples.py` | Example scaffolds | Yes (outside markers) |
| `{group}_test.py` | Test scaffolds | Yes (outside markers) |

- `__init__.py` exports are updated but VERSION remains unchanged until release
- Symbol doc pages include a "Changes" table showing unreleased modifications
- Content outside `=== OK_EDIT: pkg-ext ... ===` markers can be customized and is preserved

### Files Updated During Release

These files are updated by `post-merge` after PR is merged (main branch only):

| File | What Changes |
|------|--------------|
| `.groups.yaml` | Copied from `.groups-dev.yaml` |
| `CHANGELOG.md` | Copied from `CHANGELOG-dev.md` |
| `{pkg}/__init__.py` | VERSION updated to new version |
| `pyproject.toml` | Version field updated (if used) |
| `{pkg}.api.yaml` | Regenerated with new version |
| `docs/**/*.md` | Unreleased changes become versioned |

### File Contents Examples

**`__init__.py`:**

```python
# Generated by pkg-ext
# flake8: noqa
from my_pkg import my_group
from my_pkg.utils import parse_config

VERSION = "0.1.0"
__all__ = [
    "my_group",
    "parse_config",
]
```

**Group module (`my_group.py`):**

```python
# Generated by pkg-ext
from my_pkg.helpers import helper_func as _helper_func
from my_pkg._warnings import _experimental

helper_func = _experimental(_helper_func)  # With experimental stability
```

The underscore alias pattern prevents re-export issues with `__all__`.

**`_warnings.py`:**

When any group has non-GA stability, pkg-ext generates a `_warnings.py` module in the target package (removes runtime dependency on pkg-ext):

```python
class MyPkgWarning(UserWarning): ...
class MyPkgExperimentalWarning(MyPkgWarning): ...
class MyPkgDeprecationWarning(MyPkgWarning, DeprecationWarning): ...
```

## Symbol Detection

The tool parses Python files using AST to find:
- **Functions** - Public functions (not starting with `_`)
- **Classes** - Public classes
- **Exceptions** - Classes inheriting from `Exception` or `BaseException`
- **Type Aliases** - Names ending with `T`
- **Global Variables** - UPPERCASE names with 2+ characters

Files skipped:
- `__init__.py`, `__main__.py` (dunder files)
- `*_test.py`, `test_*.py`, `conftest.py` (test files)
- Files starting with the configured `file_header` (already generated)

## Automatic Behaviors

### Function Argument Exposure

When exposing a function, its type hint arguments are auto-exposed if they reference local package types.

### Git Integration

- Uses [GitPython](https://gitpython.readthedocs.io/) for commit analysis
- Uses [gh CLI](https://cli.github.com/) to detect PR info
- Extracts PR number from merge commit message (`Merge pull request #123`)

## API Diff and Breaking Change Detection

During `pre-commit`, pkg-ext compares `{pkg}.api.yaml` (baseline from last release) against `{pkg}.api-dev.yaml` (current code) to detect API changes.

### Detected Change Types

| Change | Breaking? | `change_kind` |
|--------|-----------|---------------|
| Parameter removed | Yes | `param_removed` |
| Required parameter added | Yes | `required_param_added` |
| Parameter type changed | Yes | `param_type_changed` |
| Return type changed | Yes | `return_type_changed` |
| Default removed | Yes | `default_removed` |
| Required field added | Yes | `required_field_added` |
| Field removed | Yes | `field_removed` |
| Base class removed | Yes | `base_class_removed` |
| Base class added | No | `base_class_added` |
| Optional parameter added | No | `optional_param_added` |
| Default added | No | `default_added` |
| Default changed | No | `default_changed` |
| Optional field added | No | `optional_field_added` |

### Auto-Generated Actions

API diff creates `BreakingChangeAction` or `AdditionalChangeAction` entries with `auto_generated: true`. These are:
- Replaced on each `pre-commit` run
- Keyed by `(name, group, type, change_kind)` for deduplication
- Timestamps preserved for unchanged changes

Interactive actions (from `pre-change`) are never replaced.

### First Release

When no baseline `{pkg}.api.yaml` exists, diff is skipped (nothing to compare against).

## Limitations

### Symbol Detection
- **Type aliases require `T` suffix** - e.g., `ConfigT` not `Config`
- **Global vars require UPPERCASE** - e.g., `DEFAULT_TIMEOUT` not `default_timeout`
- **Exceptions require `Error` suffix** - e.g., `ParseError` not `ParseException`
- **No relative import support** - Only `from pkg.module import ...` is tracked

### Group Handling
- **One group per module** - All symbols from a module belong to the same group
- **Cannot move symbols between groups** - Once assigned, module-to-group mapping is fixed
- **Root group always exists** - Cannot be removed, used for top-level exports

### Git Requirements
- **Requires `gh` CLI** for PR info detection
- **Merge commit format expected** - `Merge pull request #123 from ...`
- **Single remote assumed** - Uses first remote for URL

### Changelog
- **PR-based storage** - Each PR gets one `.yaml` file
- **No conflict resolution** - Manual merge of `.changelog/` files needed
- **Archiving by PR number** - Old entries archived to `.changelog/000/*.yaml`

### Version Bumping
- **SemVer only** - No calendar versioning support
- **`pyproject.toml` or `__init__.py`** - Version must exist in one of these
- **Pre-release suffixes** - Supports `rc`, `a` (alpha), `b` (beta)

### Interactive Mode
- **Removed reference handling incomplete** - `select_ref` and `select_multiple_ref_state` raise `NotImplementedError`. This breaks rename workflows when symbols are removed.
- **Alias creation not implemented** - `confirm_create_alias` always returns `False`

### Stability
- **Non-callable symbols** - Constants and type aliases in experimental/deprecated groups don't emit warnings. `@experimental` and `@deprecated` only work on functions and classes.
- **Arg-level only for GA groups** - Cannot track arg-level stability changes until group is GA.

### API Diff
- **No rename detection** - Renames are treated as remove + add (two separate actions)
- **Return types always breaking** - No semantic analysis (e.g., returning subclass is flagged as breaking)
- **Factory defaults** - Defaults using `"..."` (factory pattern) may cause false positives

## Dependencies

- **[ask-shell](https://github.com/EspenAlbert/ask-shell)** - Interactive prompts and shell execution
- **[model-lib](https://github.com/EspenAlbert/model-lib)** - YAML/TOML parsing and Pydantic models
- **[GitPython](https://gitpython.readthedocs.io/)** - Git repository access

## Appendix

### File Structure

```
my-repo/
  CHANGELOG.md           # Human-readable changelog
  .groups.yaml           # Group definitions
  .changelog/            # Per-PR changelog actions
    123.yaml             # Actions from PR #123
    000/                 # Archived old entries
      001.yaml
  my_pkg/
    __init__.py          # Generated exports
    my_group.py          # Generated group module
    _warnings.py         # Generated stability module (if needed)
    utils.py             # Source file
    _internal.py         # Private module (ignored)
```

### CI Configuration

**GitHub Actions:**

```yaml
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pkg-ext
      - run: pkg-ext pre-commit

  release:
    if: github.ref == 'refs/heads/main'
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: pip install pkg-ext
      - run: pkg-ext post-merge --push
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, workflow, and git hooks.