iconfig-py 0.1.10
A hierarchical configuration management system for Python applications with fast key-based lookups
License: MIT
Requires Python: >=3.10
Keywords: config, configuration, dotenv, environment, hierarchical, management, settings, yaml

# iconfig

A hierarchical configuration management system for Python applications that provides fast key-based lookups across multiple YAML configuration files.

## OBJECTIVE

`iconfig` is designed to solve the problem of managing complex, hierarchical configuration in modern applications. It provides a clean, simple interface with powerful features underneath:

- **Simple API**: Just import `iConfig` and call `config.get('key')` - that's it!
- **Hierarchical Configuration Loading**: Automatically discovers and indexes YAML files across directory structures
- **Fast Key Lookups**: Build an index once, then perform O(1) key lookups without re-parsing files
- **Singleton Pattern**: One configuration instance per application (configurable)
- **Environment Variable Expansion**: Supports `${VAR}` syntax in configuration values
- **Level-based Overrides**: Higher-level configurations can override lower-level ones
- **Path-based Filtering**: Find keys within specific configuration contexts
- **Configuration Tracing**: Use `whereis()` to find exactly where a configuration value comes from

Perfect for applications that need to merge configuration from multiple sources (system-wide, user-specific, project-specific) with predictable override behavior and a clean, intuitive API.

## INSTALLATION

### Prerequisites

- Python 3.10+
- PyYAML
- python-dotenv (optional, for .env file support)

### Install with uv (recommended)

```bash
git clone https://github.com/heinerlehr/iconfig.git
cd iconfig
uv sync --dev
uv pip install -e .
```

### Install with pip

```bash
git clone https://github.com/heinerlehr/iconfig.git
cd iconfig
pip install -e .
```

### Dependencies

```bash
pip install pyyaml python-dotenv
```

### Environment Variables

Set these environment variables to customize behavior:

- `ICONFIG_HOME`: Configuration directory (default: `"config"`)
- `ICONFIG_INDEXFN`: Index file name (default: `".index.yaml"`)


## USAGE

### Basic Usage

```python
from iconfig import iConfig

# Initialize with default config directory (or set ICONFIG_HOME)
config = iConfig()

# Get a configuration value
app_name = config.get('app_name')
database_port = config.get('port', path=['database'])

# Get with fallback default
api_timeout = config.get('timeout', default=30)

# Use as a callable (shorthand)
debug_mode = config('debug', default=False)
```

### Environment Variable Expansion

Configuration values support environment variable expansion:

```yaml
# config.yaml
database:
  url: "${DATABASE_URL}"
  host: "${DB_HOST:-localhost}"    # With default value
paths:
  data: "${HOME}/data"
  logs: "${LOG_DIR}/app.log"
```

### Keys

Search can be carried out with a string key and a list of qualifier. In order to access `key` in this structure:

```
level1:
    level2:
        level3:
            key: value
```

all of the following calls will result in `value`:

```
config = iConfig()
# All equivalent
value = config('key', path=['level1','level2','level3'])
value = config('key', 'level1', 'level2', 'level3')
value = config('level1.level2.level3.key')
```

### Overwriting and search algorithm

iConfig does not overwrite any (duplicate) key found in configuration files. Instead, it keeps all (used) configuration files separately in memory. 
With knowledge of the appropriate source, a developer can use the exact key wanted.

iConfig differentiates `depth`and `level`. 

- `level` is the hierarchy level. Top level configuration files in `ICONFIG_HOME` receive `level` 0. YAML files found in subdirectories will receive level 1 and so forth.

- `depth` is the nesting level. Within a configuration file, top-level keys have `depth` 0. Subkeys have `depth` 1 and so forth. `depth` is equivalent to the length of `path`.

- `path` is the "path" within the YAML file to the key. For the following example:
```
level1:
    level2:
        level3:
            key: value
```
the key `key` would have the path `['level1', 'level2', 'level3]` or `level1.level2.level3` in iConfig's special notation.

When not specifying `level`, `depth` and `path`, iConfig prefers
- the highest level (i.e. deepest configuration file)
- the lowest depth (i.e. the lowest nesting)

If that is not unique (i.e. several keys at the same level and depth), a `KeyError` will be thrown. Users are then encouraged to specify
the path either explicitly `['level1', 'level2', 'level3]`) or in iConfig's special notation (`level1.level2.level3`).


### Example directory Structure

```
your-project/
├── config/                    # Default config directory
│   ├── global.yaml           # System-wide settings
│   ├── app.yaml              # Application settings
│   └── modules/              # Module-specific configs
│       ├── database.yaml
│       └── api.yaml
├── .env                      # Environment variables (optional)
└── your_app.py
```

**config/global.yaml:**
```yaml
app_name: "MyApp"
version: "1.0.0"
database:
  host: "localhost"
  port: 5432
  timeout: 30
logging:
  level: "INFO"
  file: "/var/log/myapp.log"
```

**config/modules/database.yaml:**
```yaml
database:
  pool_size: 10
  timeout: 60        # Overrides global timeout for database
  ssl_enabled: true
```

### Advanced Usage

```python
import os
from iconfig import iConfig

# Use custom configuration directory
os.environ['ICONFIG_HOME'] = '/etc/myapp'
config = iConfig()

# Get values with path filtering
db_host = config.get('host', path=['database'])
api_host = config.get('host', path=['api'])

# Get values at specific hierarchy levels
global_timeout = config.get('timeout', level=0)    # Top-level files only
module_timeout = config.get('timeout', level=1)    # Module-level files only

# Get values at specific depths
root_values = config.get('debug', depth=0)         # Top-level keys only
nested_values = config.get('debug', depth=2)       # 2 levels deep

# Handle ambiguous keys
try:
    port = config.get('port')  # Returns None if not found
except Exception:
    port = config.get('port', forcefirst=True)     # Take first match

# Set configuration values in memory
config.set('database_timeout', 120)

# Find where a configuration key is defined
location = config.whereis('app_name')
print(f"app_name found in: {location}")
```

## LIMITATIONS

### Current Limitations

1. **YAML Only**: Currently only supports YAML configuration files (`.yaml`, `.yml`)
2. **Read-Heavy Optimized**: Designed for read-heavy workloads; frequent updates require index rebuilds
3. **Memory Usage**: Keeps an in-memory index of all configuration keys
4. **File System Dependency**: Requires file system access; doesn't support remote configuration stores
5. **Limited Environment Variable Syntax**: Basic `${VAR}` expansion only; `${VAR:-default}` syntax requires custom implementation

### Design Decisions

- **Lazy Loading**: Configuration files are only parsed when their values are accessed
- **Index Persistence**: Index is saved to `.index.yaml` for faster startup times
- **Automatic Rebuilds**: Index automatically rebuilds when configuration files change
- **Level vs Depth**: 
  - **Level**: File hierarchy level (0=top-level files, 1=subdirectory files)
  - **Depth**: Key nesting depth within files (0=root keys, 1=nested once)

### Performance Considerations

- Initial index building: O(n) where n = total keys across all files
- Key lookups: O(1) average case
- File watching: Uses file modification times, not real-time file system events
- Memory usage: Proportional to number of unique keys across all configuration files

## ARCHITECTURE

### Design Overview

`iconfig` uses a two-layer architecture:

- **`iConfig`**: The user-facing API that provides a clean, simple interface
- **`KeyIndex`**: The internal engine that handles file discovery, indexing, and fast lookups

This separation allows for a simple user experience while maintaining powerful functionality under the hood.

```python
# User interacts with iConfig
from iconfig import iConfig
config = iConfig()
value = config.get('key')

# iConfig internally uses KeyIndex for heavy lifting
# KeyIndex handles file scanning, indexing, YAML parsing, etc.
```

### Singleton Behavior

By default, `iConfig` uses a singleton pattern - all instances share the same configuration data. 

This can be controlled via configuration settings by including a section
```
iconfig:
    singleton: false
```

in any *.yaml in the configuration path.


## DEVELOPMENT

### Running Tests

```bash
# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=iconfig

# Run specific test file
uv run pytest tests/test_iconfig.py -v
```

### Project Structure

```
iconfig/
├── src/iconfig/
│   ├── __init__.py        # Main exports (iConfig class)
│   ├── iconfig.py         # Main iConfig user interface
│   ├── keyindex.py        # Internal KeyIndex engine
│   ├── labels.py          # String constants for keys
│   └── utils.py           # Utility functions
├── tests/
│   ├── conftest.py        # Pytest fixtures
│   ├── test_iconfig.py    # Main iConfig interface tests
│   ├── test_keyindex.py   # Internal KeyIndex tests
│   └── fixtures/          # Test configuration files
└── README.md
```

### Documentation

Full API documentation is available online and can be built locally:

**Online Documentation:**
- **Read the Docs**: [iconfig.readthedocs.io](https://iconfig.readthedocs.io/) (automatically updated)
- **GitHub Pages**: [heinerlehr.github.io/iconfig](https://heinerlehr.github.io/iconfig/) (updated on each commit)

**Local Documentation:**
```bash
# Build documentation
make build-docs

# Build and serve documentation locally
make serve-docs

# Build docs before committing (automatic via git hook)
make commit
```

The documentation includes:
- Complete API reference for all classes and methods
- Usage examples and configuration patterns
- Architecture overview and design decisions

**Documentation Features:**
- 📚 Automatic builds on every commit via GitHub Actions
- 🔧 Pre-commit hooks ensure docs build successfully
- 🌐 Published to Read the Docs and GitHub Pages
- 📖 Built from docstrings and RST files in `docs/` directory

### Contributing

1. Fork the repository
2. Create a feature branch: `git checkout -b feature-name`
3. Make changes and add tests
4. Run tests: `uv run pytest`
5. Build documentation: `make build-docs`
6. Submit a pull request

## LICENSE

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

Copyright (c) 2025 Heiner Lehr

## CHANGELOG

### v0.1.0
- Initial release
- Clean iConfig interface with simple `get()`, `set()`, `whereis()` methods
- Callable interface: `config('key')` shorthand
- Singleton pattern support (configurable)
- Basic hierarchical configuration loading
- YAML file support with anchors and aliases
- Environment variable expansion
- Index-based fast lookups with KeyIndex engine
- Configuration source tracing

## AUTHORS

- [heinerlehr](https://github.com/heinerlehr)

## SUPPORT

- **Issues**: [GitHub Issues](https://github.com/heinerlehr/iconfig/issues)
- **Documentation**: See this README and inline code documentation
- **Examples**: Check the `tests/fixtures/` directory for configuration examples