lib-layered-config 5.4.1
Cross-platform layered configuration loader for Python
License: MIT
Requires Python: >=3.10
Keywords: configuration, dotenv, env, layered, toml

# lib_layered_config

<!-- Badges -->
[![CI](https://github.com/bitranox/lib_layered_config/actions/workflows/default_cicd_public.yml/badge.svg)](https://github.com/bitranox/lib_layered_config/actions/workflows/default_cicd_public.yml)
[![CodeQL](https://github.com/bitranox/lib_layered_config/actions/workflows/codeql.yml/badge.svg)](https://github.com/bitranox/lib_layered_config/actions/workflows/codeql.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Open in Codespaces](https://img.shields.io/badge/Codespaces-Open-blue?logo=github&logoColor=white&style=flat-square)](https://codespaces.new/bitranox/lib_layered_config?quickstart=1)
[![PyPI](https://img.shields.io/pypi/v/lib-layered-config.svg)](https://pypi.org/project/lib-layered-config/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/lib-layered-config.svg)](https://pypi.org/project/lib-layered-config/)
[![Code Style: Ruff](https://img.shields.io/badge/Code%20Style-Ruff-46A3FF?logo=ruff&labelColor=000)](https://docs.astral.sh/ruff/)
[![codecov](https://codecov.io/gh/bitranox/lib_layered_config/graph/badge.svg)](https://codecov.io/gh/bitranox/lib_layered_config)
[![Maintainability](https://qlty.sh/gh/bitranox/projects/lib_layered_config/maintainability.svg)](https://qlty.sh/gh/bitranox/projects/lib_layered_config)
[![Known Vulnerabilities](https://snyk.io/test/github/bitranox/lib_layered_config/badge.svg)](https://snyk.io/test/github/bitranox/lib_layered_config)
[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)

A cross-platform configuration loader that deep-merges application defaults, host overrides, user profiles, `.env` files, and environment variables into a single immutable object. The core follows Clean Architecture boundaries so adapters (filesystem, dotenv, environment) stay isolated from the domain model while the CLI mirrors the same orchestration.

## Table of Contents

### Getting Started
- [Key Features](#key-features)
- [Installation](#installation)
- [Quick Start](#quick-start)

### Core Concepts
- [Understanding Key Identifiers](#understanding-key-identifiers-vendor-app-slug-and-profile) — Vendor, App, Slug, Profile
- [Configuration Profiles](#configuration-profiles)
- [Configuration File Structure](#configuration-file-structure)
- [Configuration Sources & Precedence](#configuration-sources--precedence)

### Command Line Interface
- [CLI Command Summary](#command-summary)
- [`read` / `read-json`](#read) — Load and inspect configuration
- [`deploy`](#deploy) — Deploy configuration files
- [`generate-examples`](#generate-examples) — Scaffold example trees
- [`env-prefix`](#env-prefix) — Compute environment prefix
- [`info` / `fail`](#info) — Diagnostics

### Python API
- [Layer Enum](#layer-enum)
- [Config Class](#config-class)
- [`read_config`](#read_config) / [`read_config_json`](#read_config_json) / [`read_config_raw`](#read_config_raw)
- [`deploy_config`](#deploy_config)
- [`generate_examples` (Python)](#generate_examples)
- [`default_env_prefix`](#default_env_prefix)
- [Profile Validation](#profile-validation)
- [Permission Constants](#permission-constants)

### Deployment & Security
- [File Permissions](#-file-permissions)
- [Security Best Practices](#-security-best-practices)
- [Recommendations for Sensitive Data](#-recommendations-for-sensitive-data)
- [Example Generation & Deployment](#example-generation--deployment)
- [Deploying with `.d` Directories](#deploying-with-d-directories)
- [User Files Preservation](#user-files-are-preserved-during-deployment)

### Reference
- [Provenance & Observability](#provenance--observability)
- [Architecture Overview](#architecture-overview)
- [Further Documentation](#further-documentation)
- [Development](#development)
- [License](#license)

## Key Features

- **Deterministic layering** — precedence is always `defaults → app → host → user → dotenv → env`.
- **Immutable value object** — returned `Config` prevents accidental mutation and exposes dotted-path helpers.
- **Provenance tracking** — every key reports the layer and path that produced it.
- **Cross-platform path discovery** — Linux (XDG), macOS, and Windows layouts with environment overrides for tests.
- **Configuration profiles** — organize environment-specific configs (test, staging, production) into isolated subdirectories.
- **`.d` directory support** — split configuration into multiple files using the Linux `.d` pattern (e.g., `config.d/10-database.toml`). Supports mixed formats (TOML, YAML, JSON) in the same directory.
- **Easy deployment** — deploy configs to app, host, and user layers with smart conflict handling that protects user customizations through automatic backups (`.bak`) and UCF files (`.ucf`) for safe CI/CD updates.
- **Fast parsing** — uses `rtoml` (Rust-based) for ~5x faster TOML parsing than stdlib `tomllib`.
- **Extensible formats** — TOML and JSON are built-in; YAML is available via the optional `yaml` extra.
- **Automation-friendly CLI** — inspect, deploy, or scaffold configurations without writing Python.
- **Structured logging** — adapters emit trace-aware events without polluting the domain layer.

## Installation

```bash
pip install lib_layered_config
# or with optional YAML support
pip install "lib_layered_config[yaml]"
```

> **Requires Python 3.10+** — uses `rtoml` (Rust-based TOML parser) for ~5x faster parsing than stdlib `tomllib`.
>
> Install the optional `yaml` extra only when you actually ship `.yml` files to keep the dependency footprint small.

For local development add tooling extras:

```bash
pip install "lib_layered_config[dev]"
```

## Quick Start

```python
from lib_layered_config import read_config

config = read_config(vendor="Acme", app="ConfigKit", slug="config-kit")
print(config.get("service.timeout", default=30))
print(config.origin("service.timeout"))
```

CLI equivalent (human readable by default):

```bash
lib_layered_config read --vendor Acme --app ConfigKit --slug config-kit
```

JSON output including provenance:

```bash
lib_layered_config read --vendor Acme --app ConfigKit --slug config-kit --format json
# or
lib_layered_config read-json --vendor Acme --app ConfigKit --slug config-kit
```

## Understanding Key Identifiers: Vendor, App, Slug, and Profile

Before diving into configuration sources, it's important to understand the four key identifiers used throughout this library:

### Vendor

**What it is:** Your organization or company name (e.g., `"Acme"`, `"Mozilla"`, `"MyCompany"`).

**Where it's used:**
- **macOS:** `/Library/Application Support/Acme/MyApp/` and `~/Library/Application Support/Acme/MyApp/`
- **Windows:** `C:\ProgramData\Acme\MyApp\` and `%APPDATA%\Acme\MyApp\`
- **Linux:** Not used (Linux uses the slug directly)

**Example:**
```python
# Your company is "Acme Corp"
config = read_config(vendor="Acme", app="DatabaseTool", slug="db-tool")
# macOS paths: /Library/Application Support/Acme/DatabaseTool/config.toml
```

---

### App

**What it is:** Your application's full/display name (e.g., `"DatabaseTool"`, `"ConfigKit"`, `"MyService"`).

**Where it's used:**
- **macOS:** Combined with vendor in paths: `/Library/Application Support/Acme/MyApp/`
- **Windows:** Combined with vendor: `C:\ProgramData\Acme\MyApp\`
- **Linux:** Not used (Linux uses the slug directly)
- **Default slug:** If you don't specify a slug, the app name is used as the slug

**Example:**
```python
config = read_config(vendor="Acme", app="ConfigKit", slug="config-kit")
# macOS: /Library/Application Support/Acme/ConfigKit/config.toml
# Windows: C:\ProgramData\Acme\ConfigKit\config.toml
```

---

### Slug (Configuration Slug)

**What it is:** A lowercase, filesystem-friendly identifier for your configuration (e.g., `"myapp"`, `"config-kit"`, `"db-tool"`).

**Why it exists:** The slug serves as a **universal, platform-independent identifier** for your configuration that works consistently across:
1. Linux/UNIX filesystem paths (case-sensitive, prefers hyphens)
2. Environment variable prefixes (converted to uppercase)
3. Cross-platform scripts and automation

**Where it's used:**

#### 1. **Linux/UNIX Paths**
```bash
/etc/xdg/myapp/config.toml                    # System-wide (XDG-compliant)
/etc/xdg/myapp/config.d/10-database.toml      # Optional split config
/etc/xdg/myapp/config.d/20-logging.toml       # Files merged in order
/etc/xdg/myapp/hosts/server-01.toml           # Host-specific (XDG-compliant)
~/.config/myapp/config.toml                   # User-specific
~/.config/myapp/config.d/90-local.toml        # User's split config
~/.config/myapp/.env                          # Environment variables
```

Note: For backwards compatibility, the library also checks `/etc/myapp/` if `/etc/xdg/myapp/` is not found.

#### 2. **Environment Variable Prefix**
The slug is converted to uppercase with underscores, followed by a triple underscore (`___`) separator to clearly distinguish the prefix from section/key separators (which use double underscores `__`):
```bash
# Slug: "myapp" → Environment prefix: "MYAPP___"
MYAPP___DATABASE__HOST=localhost
MYAPP___DATABASE__PORT=5432
MYAPP___SERVICE__TIMEOUT=30

# Slug: "config-kit" → Environment prefix: "CONFIG_KIT___"
CONFIG_KIT___API__KEY=secret
CONFIG_KIT___DEBUG__ENABLED=true
```

#### 3. **Cross-Platform Consistency**
The slug provides a consistent identifier regardless of platform:
```python
# Same slug works on all platforms
config = read_config(vendor="Acme", app="My App", slug="myapp")

# Linux:   /etc/xdg/myapp/config.toml (+ optional config.d/)
# macOS:   /Library/Application Support/Acme/My App/config.toml (+ optional config.d/)
# Windows: C:\ProgramData\Acme\My App\config.toml (+ optional config.d\)
# Env vars: MYAPP___DATABASE__HOST (all platforms)
```

---

### Slug Naming Best Practices

✅ **DO:**
- Use lowercase letters: `"myapp"`, `"database-tool"`
- Use hyphens for word separation: `"config-kit"`, `"db-manager"`
- Keep it short and memorable: `"myapp"` not `"my-super-awesome-application"`
- Use ASCII characters only: `"myapp"` not `"my-àpp"`
- Use the same slug everywhere in your application

❌ **DON'T:**
- Use spaces: `"my app"` → use `"myapp"` or `"my-app"`
- Use uppercase: `"MyApp"` → use `"myapp"` (uppercase works but isn't recommended)
- Use underscores in the slug: `"my_app"` → use `"my-app"` (underscores are added automatically for env vars)
- Use non-ASCII characters: `"café"` → will raise `ValueError`
- Use Windows reserved names: `"CON"`, `"PRN"`, `"NUL"` → will raise `ValueError`
- Mix naming conventions across your codebase
- Use path separators (`/` or `\`): `"../etc"` will raise `ValueError`
- Start with a dot: `".hidden"` will raise `ValueError`

---

### Profile (Optional)

**What it is:** An optional identifier for environment-specific configurations (e.g., `"test"`, `"staging"`, `"production"`).

**Why it exists:** Profiles allow you to organize separate configuration sets for different environments (development, testing, staging, production) without mixing files or relying solely on environment variables.

**Where it's used:**
When a profile is specified, a `profile/<name>/` subdirectory is inserted into all configuration paths:

#### 1. **Linux/UNIX Paths (with profile)**
```bash
# Without profile:
/etc/xdg/myapp/config.toml
/etc/xdg/myapp/config.d/10-database.toml           # Optional split config
/etc/xdg/myapp/config.d/20-logging.toml
~/.config/myapp/config.toml
~/.config/myapp/config.d/90-local.toml             # User overrides

# With profile="production":
/etc/xdg/myapp/profile/production/config.toml
/etc/xdg/myapp/profile/production/config.d/10-database.toml
/etc/xdg/myapp/profile/production/config.d/20-cache.toml
~/.config/myapp/profile/production/config.toml
~/.config/myapp/profile/production/config.d/90-local.toml
```

#### 2. **macOS Paths (with profile)**
```bash
# Without profile:
/Library/Application Support/Acme/MyApp/config.toml
/Library/Application Support/Acme/MyApp/config.d/10-database.toml

# With profile="production":
/Library/Application Support/Acme/MyApp/profile/production/config.toml
/Library/Application Support/Acme/MyApp/profile/production/config.d/10-database.toml
```

#### 3. **Windows Paths (with profile)**
```bash
# Without profile:
C:\ProgramData\Acme\MyApp\config.toml
C:\ProgramData\Acme\MyApp\config.d\10-database.toml

# With profile="production":
C:\ProgramData\Acme\MyApp\profile\production\config.toml
C:\ProgramData\Acme\MyApp\profile\production\config.d\10-database.toml
```

#### 4. **Usage Example**
```python
from lib_layered_config import read_config

# Load production configuration
prod_config = read_config(
    vendor="Acme",
    app="MyApp",
    slug="myapp",
    profile="production"
)

# Load test configuration (different paths, completely isolated)
test_config = read_config(
    vendor="Acme",
    app="MyApp",
    slug="myapp",
    profile="test"
)

# Load default configuration (no profile, original paths)
default_config = read_config(
    vendor="Acme",
    app="MyApp",
    slug="myapp"
    # profile=None (default)
)
```

#### 5. **CLI Usage**
```bash
# Read production profile
lib_layered_config read --vendor Acme --app MyApp --slug myapp --profile production

# Deploy to test profile
lib_layered_config deploy --source config.toml --vendor Acme --app MyApp --slug myapp --profile test --target app
```

---

### Profile Naming Best Practices

✅ **DO:**
- Use lowercase letters: `"test"`, `"production"`
- Use hyphens for word separation: `"staging-v2"`, `"dev-local"`
- Keep it short and descriptive: `"prod"` or `"production"`
- Use consistent profile names across your infrastructure

❌ **DON'T:**
- Use spaces: `"my profile"` → use `"my-profile"`
- Use non-ASCII characters: `"tëst"` → will raise `ValueError`
- Use Windows reserved names: `"CON"`, `"NUL"` → will raise `ValueError`
- Use path separators: `"../etc"` → will raise `ValueError`
- Exceed 256 characters (absolute limit for filesystem safety)

---

### Complete Example: How They Work Together

```python
from lib_layered_config import read_config

# Define your application identity (without profile)
config = read_config(
    vendor="Acme",           # Your company name
    app="DatabaseManager",   # Your application's display name
    slug="db-manager"        # Filesystem/environment-friendly identifier
)

# Or with a profile for environment-specific configuration
prod_config = read_config(
    vendor="Acme",
    app="DatabaseManager",
    slug="db-manager",
    profile="production"     # Optional: isolates config in profile subdirectory
)
```

**This creates the following structure (without profile):**

**On Linux:**
```
/etc/xdg/db-manager/config.toml               # System-wide (uses slug, XDG-compliant)
/etc/xdg/db-manager/config.d/10-connection.toml   # Split config (optional)
/etc/xdg/db-manager/config.d/20-pools.toml
~/.config/db-manager/config.toml              # User-specific (uses slug)
~/.config/db-manager/config.d/90-local.toml   # User overrides (optional)
Environment: DB_MANAGER___*                   # Env prefix (slug → uppercase + ___)
```

**On macOS:**
```
/Library/Application Support/Acme/DatabaseManager/config.toml
/Library/Application Support/Acme/DatabaseManager/config.d/10-connection.toml
~/Library/Application Support/Acme/DatabaseManager/config.toml
~/Library/Application Support/Acme/DatabaseManager/config.d/90-local.toml
Environment: DB_MANAGER___*
```

**On Windows:**
```
C:\ProgramData\Acme\DatabaseManager\config.toml
C:\ProgramData\Acme\DatabaseManager\config.d\10-connection.toml
%APPDATA%\Acme\DatabaseManager\config.toml
%APPDATA%\Acme\DatabaseManager\config.d\90-local.toml
Environment: DB_MANAGER___*
```

**With `profile="production"`:**

| Platform | Path (+ optional config.d/) |
|----------|------|
| Linux | `/etc/xdg/db-manager/profile/production/config.toml` |
| macOS | `/Library/Application Support/Acme/DatabaseManager/profile/production/config.toml` |
| Windows | `C:\ProgramData\Acme\DatabaseManager\profile\production\config.toml` |

---

### Why Four Identifiers?

**Different platforms have different conventions:**

- **Windows/macOS:** Prefer human-readable names with spaces and mixed case (`"Acme Corp"`, `"My Application"`)
- **Linux/UNIX:** Prefer lowercase with hyphens (`myapp`, `config-kit`)
- **Environment variables:** Must use uppercase with underscores (`MYAPP_`, `CONFIG_KIT_`)
- **Profiles:** Allow environment-specific configuration isolation (`test`, `staging`, `production`)

This library uses four identifiers so your application can follow **native conventions on each platform** while maintaining a **consistent configuration identity** and supporting **environment-specific configurations**.

---

### Quick Reference Table

| Identifier  | Format                               | Example                     | Used In                                                |
|-------------|--------------------------------------|-----------------------------|--------------------------------------------------------|
| **vendor**  | ASCII, spaces allowed                | `"Acme"`, `"Acme Corp"`     | macOS, Windows paths                                   |
| **app**     | ASCII, spaces allowed                | `"My App"`, `"Btx Fix Mcp"` | macOS, Windows paths                                   |
| **slug**    | lowercase-with-hyphens (recommended) | `"db-manager"`              | Linux paths, env var prefix (becomes `DB_MANAGER___`)  |
| **profile** | lowercase-with-hyphens (recommended) | `"production"`              | Optional subdirectory for environment-specific configs |

**All identifiers are validated** to ensure cross-platform filesystem safety. See [Identifier Validation Rules](#identifier-validation-rules) below.

---

### Configuration Profiles

Profiles allow you to organize environment-specific configurations (e.g., `test`, `staging`, `production`) into isolated subdirectories. When a profile is specified, all configuration paths include a `profile/<name>/` segment.

#### How Profiles Work

**Without profile:**
```
/etc/xdg/myapp/config.toml
/etc/xdg/myapp/config.d/10-database.toml              # Optional split config
/etc/xdg/myapp/config.d/20-logging.toml
/etc/xdg/myapp/hosts/server-01.toml
~/.config/myapp/config.toml
~/.config/myapp/config.d/90-local.toml                # User overrides
```

**With `profile="production"`:**
```
/etc/xdg/myapp/profile/production/config.toml
/etc/xdg/myapp/profile/production/config.d/10-database.toml
/etc/xdg/myapp/profile/production/config.d/20-cache.toml
/etc/xdg/myapp/profile/production/hosts/server-01.toml
~/.config/myapp/profile/production/config.toml
~/.config/myapp/profile/production/config.d/90-local.toml
```

#### Using Profiles in Python

```python
from lib_layered_config import read_config

# Load production configuration
config = read_config(
    vendor="Acme",
    app="ConfigKit",
    slug="config-kit",
    profile="production"
)

# Load test configuration
test_config = read_config(
    vendor="Acme",
    app="ConfigKit",
    slug="config-kit",
    profile="test"
)
```

#### Using Profiles in CLI

```bash
# Read configuration for production profile
lib_layered_config read --vendor Acme --app ConfigKit --slug config-kit --profile production

# Deploy configuration to production profile paths
lib_layered_config deploy --source config.toml --vendor Acme --app ConfigKit --slug config-kit --profile production --target app
```

#### Profile Path Examples

Each path can have an optional companion `.d` directory (e.g., `config.d/`) for split configuration.

| Platform          | Without Profile                             | With `profile="test"`                                  |
|-------------------|---------------------------------------------|--------------------------------------------------------|
| **Linux (app)**   | `/etc/xdg/<slug>/config.toml`               | `/etc/xdg/<slug>/profile/test/config.toml`             |
| **Linux (host)**  | `/etc/xdg/<slug>/hosts/<hostname>.toml`     | `/etc/xdg/<slug>/profile/test/hosts/<hostname>.toml`   |
| **Linux (user)**  | `~/.config/<slug>/config.toml`              | `~/.config/<slug>/profile/test/config.toml`            |
| **macOS (app)**   | `/Library/.../<vendor>/<app>/config.toml`   | `/Library/.../<vendor>/<app>/profile/test/config.toml` |
| **Windows (app)** | `C:\ProgramData\<vendor>\<app>\config.toml` | `C:\ProgramData\...\profile\test\config.toml`          |

#### Profile Naming Rules

Profile names follow the same validation as other identifiers (see below), with an additional length constraint:
- **Default max length:** 64 characters (configurable via `max_profile_length`)
- **Absolute max length:** 256 characters (security hardening, cannot be overridden)

**Valid:** `test`, `production`, `staging-v2`, `dev_local`
**Invalid:** `../etc`, `.hidden`, `my profile`, `CON`, names exceeding 256 characters

---

### Identifier Validation Rules

All identifiers are validated to ensure they are safe for use as filesystem directory names on both Windows and Linux.

#### Validation by Identifier Type

| Identifier   | Spaces Allowed | Used For                                                         |
|--------------|----------------|------------------------------------------------------------------|
| **vendor**   | ✅ Yes          | macOS/Windows paths (`/Library/Application Support/Acme Corp/`)  |
| **app**      | ✅ Yes          | macOS/Windows paths (`/Library/Application Support/.../My App/`) |
| **slug**     | ❌ No           | Linux paths, environment variable prefix                         |
| **profile**  | ❌ No           | Profile subdirectory name                                        |
| **hostname** | ❌ No           | Host-specific config files                                       |

#### Common Validation Rules (All Identifiers)

| Rule                             | Description                                         | Example Invalid Value             |
|----------------------------------|-----------------------------------------------------|-----------------------------------|
| **ASCII-only**                   | No Unicode/UTF-8 special characters                 | `café`, `日本語`, `app🚀`            |
| **Must start with alphanumeric** | Cannot start with dot, hyphen, underscore, or space | `.hidden`, `-app`, `_private`     |
| **No path separators**           | Prevents path traversal attacks                     | `../etc`, `foo/bar`, `C:\Windows` |
| **No Windows-invalid chars**     | `<`, `>`, `:`, `"`, `\|`, `?`, `*` are forbidden    | `app<test>`, `file:name`          |
| **No Windows reserved names**    | CON, PRN, AUX, NUL, COM1-9, LPT1-9                  | `CON`, `prn`, `NUL.txt`           |
| **Cannot end with dot/space**    | Windows restriction                                 | `app.`, `name `                   |

#### Examples

```python
from lib_layered_config import read_config

# ✅ Valid identifiers
config = read_config(
    vendor="Acme Corp",      # OK: spaces allowed in vendor
    app="Btx Fix Mcp",       # OK: spaces allowed in app
    slug="db-manager",       # OK: lowercase with hyphens (no spaces)
    profile="production"     # OK: lowercase (no spaces)
)

# ❌ These will raise ValueError
read_config(vendor="../etc", ...)      # Path traversal
read_config(app="café", ...)           # Non-ASCII character
read_config(slug="CON", ...)           # Windows reserved name
read_config(slug="my slug", ...)       # Slug cannot have spaces
read_config(profile="my profile", ...) # Profile cannot have spaces
read_config(vendor=".hidden", ...)     # Starts with dot
read_config(app="app<test>", ...)      # Windows-invalid character
```

---

## Configuration File Structure

Configuration files use TOML, JSON, or YAML format. Here's a comprehensive example showing how to structure your configuration:

### Example Configuration File (TOML)

```toml
# config.toml - Complete example showing all structural features

# ============================================================================
# TOP-LEVEL KEYS (Simple Values)
# ============================================================================
# Access in Python: config.get("debug")
# Access in CLI: config["debug"]
# Environment variable: MYAPP___DEBUG=true

debug = false
environment = "production"
version = "1.0.0"


# ============================================================================
# SECTIONS (Tables in TOML)
# ============================================================================
# Sections group related configuration values
# Access in Python: config.get("database.host")
# Environment variable: MYAPP___DATABASE__HOST=localhost

[database]
host = "localhost"
port = 5432
name = "myapp_db"
username = "admin"
# Passwords should come from environment variables or .env files!
# Set via: MYAPP___DATABASE__PASSWORD=secret


# ============================================================================
# NESTED SECTIONS (Subtables)
# ============================================================================
# Use dot notation for nested sections
# Access in Python: config.get("database.pool.size")
# Environment variable: MYAPP___DATABASE__POOL__SIZE=20

[database.pool]
size = 10
max_overflow = 20
timeout = 30
recycle = 3600  # seconds


[database.ssl]
enabled = true
verify = true
cert_path = "/etc/ssl/certs/db.pem"


# ============================================================================
# ARRAYS (Lists)
# ============================================================================
# Access in Python: config.get("database.replicas")
# Returns: ["replica1.db.local", "replica2.db.local"]

[database]
replicas = [
    "replica1.db.local",
    "replica2.db.local",
    "replica3.db.local"
]


# ============================================================================
# SERVICE CONFIGURATION
# ============================================================================

[service]
name = "MyApp Service"
host = "0.0.0.0"
port = 8080
timeout = 30
base_url = "https://api.example.com"


# Nested retry configuration
[service.retry]
max_attempts = 3
backoff_multiplier = 2
initial_delay = 1.0  # seconds


# Multiple endpoints
[service.endpoints]
api = "/api/v1"
health = "/health"
metrics = "/metrics"


# ============================================================================
# LOGGING CONFIGURATION
# ============================================================================

[logging]
level = "INFO"  # DEBUG, INFO, WARNING, ERROR, CRITICAL
format = "json"
output = "stdout"


[logging.handlers]
console = true
file = true
syslog = false


[logging.files]
path = "/var/log/myapp/app.log"
max_bytes = 10485760  # 10 MB
backup_count = 5


# ============================================================================
# FEATURE FLAGS
# ============================================================================
# Use sections for grouping related feature flags

[features]
new_ui = false
experimental_api = false
beta_features = false


[features.analytics]
enabled = true
sampling_rate = 0.1  # 10% of requests


# ============================================================================
# API CONFIGURATION
# ============================================================================

[api]
rate_limit = 1000  # requests per minute
timeout = 60


[api.authentication]
type = "jwt"  # jwt, oauth, apikey
token_expiry = 3600  # seconds


[api.cors]
enabled = true
allowed_origins = ["https://example.com", "https://app.example.com"]
allowed_methods = ["GET", "POST", "PUT", "DELETE"]


# ============================================================================
# CACHE CONFIGURATION
# ============================================================================

[cache]
backend = "redis"  # redis, memcached, memory
default_ttl = 300  # seconds


[cache.redis]
host = "localhost"
port = 6379
db = 0
password = ""  # Set via MYAPP___CACHE__REDIS__PASSWORD


# ============================================================================
# EMAIL CONFIGURATION
# ============================================================================

[email]
enabled = true
from_address = "noreply@example.com"
from_name = "MyApp"


[email.smtp]
host = "smtp.gmail.com"
port = 587
use_tls = true
username = "notifications@example.com"
# Password should come from environment: MYAPP___EMAIL__SMTP__PASSWORD


# ============================================================================
# MONITORING & OBSERVABILITY
# ============================================================================

[monitoring]
enabled = true


[monitoring.metrics]
backend = "prometheus"
port = 9090
path = "/metrics"


[monitoring.tracing]
enabled = true
backend = "jaeger"
sample_rate = 0.01  # 1% of requests


[monitoring.healthcheck]
enabled = true
interval = 30  # seconds
```

---

### Understanding the Structure

#### 1. **Top-Level Keys**
Simple key-value pairs at the root level:
```toml
debug = false
environment = "production"
```
**Access:**
- Python: `config.get("debug")` or `config["debug"]`
- CLI: Value appears as `debug: false`
- Environment: `MYAPP___DEBUG=true`

---

#### 2. **Sections (Tables)**
Sections group related configuration:
```toml
[database]
host = "localhost"
port = 5432
```
**Access:**
- Python: `config.get("database.host")` → `"localhost"`
- Python: `config["database"]` → `{"host": "localhost", "port": 5432}`
- Environment: `MYAPP___DATABASE__HOST=postgres.local`

---

#### 3. **Nested Sections (Subtables)**
Use dot notation for deeper nesting:
```toml
[database.pool]
size = 10
timeout = 30

[database.ssl]
enabled = true
verify = true
```
**Access:**
- Python: `config.get("database.pool.size")` → `10`
- Python: `config.get("database.ssl.enabled")` → `true`
- Environment: `MYAPP___DATABASE__POOL__SIZE=20`
- Environment: `MYAPP___DATABASE__SSL__ENABLED=false`

**Structure visualization:**
```python
{
    "database": {
        "host": "localhost",
        "port": 5432,
        "pool": {
            "size": 10,
            "timeout": 30
        },
        "ssl": {
            "enabled": true,
            "verify": true
        }
    }
}
```

---

#### 4. **Arrays (Lists)**
Multiple values in a list:
```toml
[database]
replicas = [
    "replica1.db.local",
    "replica2.db.local"
]

[api.cors]
allowed_origins = ["https://example.com", "https://app.example.com"]
```
**Access:**
- Python: `config.get("database.replicas")` → `["replica1.db.local", "replica2.db.local"]`
- Python: `config.get("database.replicas")[0]` → `"replica1.db.local"`

---

#### 5. **Data Types**

TOML supports multiple data types:

```toml
# Strings
name = "MyApp"
host = "localhost"

# Integers
port = 8080
timeout = 30

# Floats
sample_rate = 0.1
backoff_multiplier = 2.5

# Booleans
debug = false
enabled = true

# Arrays
replicas = ["host1", "host2"]
allowed_methods = ["GET", "POST"]

# Dates/Times (TOML feature)
created_at = 2024-01-15T10:30:00Z
```

---

### Complete Python Access Example

Given the example configuration above, here's how to access values:

```python
from lib_layered_config import read_config

config = read_config(vendor="Acme", app="MyApp", slug="myapp")

# Top-level keys
debug = config.get("debug")  # false
env = config.get("environment")  # "production"

# Section values
db_host = config.get("database.host")  # "localhost"
db_port = config.get("database.port")  # 5432

# Nested section values
pool_size = config.get("database.pool.size")  # 10
ssl_enabled = config.get("database.ssl.enabled")  # true

# Deep nesting
retry_attempts = config.get("service.retry.max_attempts")  # 3
smtp_port = config.get("email.smtp.port")  # 587

# Arrays
replicas = config.get("database.replicas")  # ["replica1.db.local", ...]
first_replica = config.get("database.replicas")[0]  # "replica1.db.local"

# Feature flags
new_ui_enabled = config.get("features.new_ui")  # false
analytics_rate = config.get("features.analytics.sampling_rate")  # 0.1

# With defaults
api_timeout = config.get("api.timeout", default=60)  # 60
cache_ttl = config.get("cache.default_ttl", default=300)  # 300
```

---

### Environment Variable Mapping

The slug is converted to an environment prefix (uppercase with triple underscore `___` separator), and nested keys use double underscores (`__`):

```bash
# Slug: "myapp" → Prefix: "MYAPP___"

# Top-level keys
MYAPP___DEBUG=true
MYAPP___ENVIRONMENT=staging

# Section keys (triple underscore after prefix, double for nesting)
MYAPP___DATABASE__HOST=postgres.production.local
MYAPP___DATABASE__PORT=5433

# Nested sections (each level separated by __)
MYAPP___DATABASE__POOL__SIZE=50
MYAPP___DATABASE__SSL__ENABLED=true

# Deep nesting
MYAPP___SERVICE__RETRY__MAX_ATTEMPTS=5
MYAPP___EMAIL__SMTP__PASSWORD=secret123
MYAPP___FEATURES__ANALYTICS__SAMPLING_RATE=0.5

# Arrays (JSON format for complex types)
MYAPP___DATABASE__REPLICAS='["replica1", "replica2"]'
```

**Key Pattern:**
- Prefix: `SLUG` in uppercase followed by `___`
- Separator: Triple underscore (`___`) after prefix to distinguish from nesting
- Nesting: Double underscores (`__`) for each level
- Format: `PREFIX___SECTION__SUBSECTION__KEY=value`

---

### JSON and YAML Equivalents

The same structure in JSON:

```json
{
  "debug": false,
  "environment": "production",
  "database": {
    "host": "localhost",
    "port": 5432,
    "pool": {
      "size": 10,
      "timeout": 30
    },
    "ssl": {
      "enabled": true,
      "verify": true
    },
    "replicas": ["replica1.db.local", "replica2.db.local"]
  },
  "service": {
    "host": "0.0.0.0",
    "port": 8080,
    "retry": {
      "max_attempts": 3,
      "backoff_multiplier": 2
    }
  }
}
```

And in YAML:

```yaml
debug: false
environment: production

database:
  host: localhost
  port: 5432
  pool:
    size: 10
    timeout: 30
  ssl:
    enabled: true
    verify: true
  replicas:
    - replica1.db.local
    - replica2.db.local

service:
  host: 0.0.0.0
  port: 8080
  retry:
    max_attempts: 3
    backoff_multiplier: 2
```

All three formats produce the same configuration structure and can be accessed identically through the library.

---

## Configuration Sources & Precedence

Later layers override earlier ones **per key** while leaving unrelated keys untouched.

| Precedence | Layer      | Description                                                           |
|------------|------------|-----------------------------------------------------------------------|
| 0          | `defaults` | Optional baseline file provided via the API/CLI `--default-file` flag |
| 1          | `app`      | System-wide defaults (e.g. `/etc/<slug>/…`)                           |
| 2          | `host`     | Machine-specific overrides (`hosts/<hostname>.toml`)                  |
| 3          | `user`     | Per-user settings (XDG, Application Support, AppData)                 |
| 4          | `dotenv`   | First `.env` found via upward search plus platform extras             |
| 5          | `env`      | Process environment with namespacing and `__` nesting                 |

Use the optional defaults layer when you want one explicitly-provided file to seed configuration before host/user overrides apply.

Important directories (overridable via environment variables):

### Linux
- `/etc/xdg/<slug>/config.toml` (XDG system-wide, checked first)
- `/etc/xdg/<slug>/config.d/*.{toml,json,yaml,yml}`
- `/etc/<slug>/config.toml` (legacy fallback)
- `/etc/<slug>/config.d/*.{toml,json,yaml,yml}`
- `/etc/xdg/<slug>/hosts/<hostname>.toml` or `/etc/<slug>/hosts/<hostname>.toml`
- `/etc/xdg/<slug>/hosts/<hostname>.d/*.{toml,json,yaml,yml}` (host-specific split config)
- `$XDG_CONFIG_HOME/<slug>/config.toml` (user; falls back to `~/.config/<slug>/config.toml`)
- `$XDG_CONFIG_HOME/<slug>/config.d/*.{toml,json,yaml,yml}`
- `.env` search: current directory upwards + `$XDG_CONFIG_HOME/<slug>/.env`

### macOS
- `/Library/Application Support/<Vendor>/<App>/config.toml` (system-wide app layer)
- `/Library/Application Support/<Vendor>/<App>/config.d/*.{toml,json,yaml,yml}`
- `/Library/Application Support/<Vendor>/<App>/hosts/<hostname>.toml`
- `/Library/Application Support/<Vendor>/<App>/hosts/<hostname>.d/*.{toml,json,yaml,yml}`
- `~/Library/Application Support/<Vendor>/<App>/config.toml` (user layer)
- `~/Library/Application Support/<Vendor>/<App>/config.d/*.{toml,json,yaml,yml}`
- `.env` search: current directory upwards + `~/Library/Application Support/<Vendor>/<App>/.env`

### Windows
- `%ProgramData%\<Vendor>\<App>\config.toml` (system-wide app layer)
- `%ProgramData%\<Vendor>\<App>\config.d\*.{toml,json,yaml,yml}`
- `%ProgramData%\<Vendor>\<App>\hosts\%COMPUTERNAME%.toml`
- `%ProgramData%\<Vendor>\<App>\hosts\%COMPUTERNAME%.d\*.{toml,json,yaml,yml}`
- `%APPDATA%\<Vendor>\<App>\config.toml` (user layer; resolver order: `LIB_LAYERED_CONFIG_APPDATA` → `%APPDATA%`; falls back to `%LOCALAPPDATA%`)
- `%APPDATA%\<Vendor>\<App>\config.d\*.{toml,json,yaml,yml}`
- `.env` search: current directory upwards + `%APPDATA%\<Vendor>\<App>\.env`

Environment overrides: `LIB_LAYERED_CONFIG_ETC`, `LIB_LAYERED_CONFIG_PROGRAMDATA`, `LIB_LAYERED_CONFIG_APPDATA`, `LIB_LAYERED_CONFIG_LOCALAPPDATA`, `LIB_LAYERED_CONFIG_MAC_APP_ROOT`, `LIB_LAYERED_CONFIG_MAC_HOME_ROOT`. Both the runtime readers and the `deploy` helper honour these variables so generated files land in the same directories that `read_config` inspects.

**Fallback note:** Whenever a path is marked as a fallback, the resolver first consults the documented environment overrides (`LIB_LAYERED_CONFIG_*`, `$XDG_CONFIG_HOME`, `%APPDATA%`, etc.). If those variables are unset or the computed directory does not exist, it switches to the stated fallback location (`~/.config`, `%LOCALAPPDATA%`, ...). This keeps local installs working without additional environment configuration while still allowing operators to steer resolution explicitly.

### The `.d` Directory Pattern

Any configuration file can have a companion `.d` directory for split configuration. This follows the common Linux pattern (similar to `/etc/apt/sources.list.d/` or `/etc/sudoers.d/`).

**Naming convention:** The `.d` directory name is the filename without extension plus `.d`:
- `config.toml` → `config.d/`
- `defaults.toml` → `defaults.d/`
- `myapp.yaml` → `myapp.d/`
- `settings.json` → `settings.d/`

This means **all formats share the same `.d` directory** - `config.toml`, `config.yaml`, and `config.json` all use `config.d/`.

**How it works:**
1. The loader first loads the base file (e.g., `config.toml`) if present
2. Then loads all files from the `.d` directory (e.g., `config.d/`) in **lexicographic order**
3. Only files with supported extensions are loaded: `.toml`, `.json`, `.yaml`, `.yml`
4. Files are merged in order, so later files override earlier ones
5. **Both the base file and `.d` directory are optional** - either can exist independently

**Supported at all layers:** The `.d` directory pattern works for **app**, **host**, and **user** layers:

| Layer    | Base File                             | Companion `.d` Directory            |
|----------|---------------------------------------|-------------------------------------|
| **app**  | `/etc/xdg/myapp/config.toml`          | `/etc/xdg/myapp/config.d/`          |
| **host** | `/etc/xdg/myapp/hosts/server-01.toml` | `/etc/xdg/myapp/hosts/server-01.d/` |
| **user** | `~/.config/myapp/config.toml`         | `~/.config/myapp/config.d/`         |

**Host-specific `.d` directories:** Each hostname file can have its own `.d` directory for per-host split configuration:
```
/etc/xdg/myapp/hosts/
├── server-01.toml                    # Host-specific base config
├── server-01.d/                      # Host-specific split config
│   ├── 10-network.toml
│   └── 20-storage.toml
├── server-02.toml
└── server-02.