project-toolkit 1.0.4
A modular toolkit for managing project paths, cloud services, and configuration in Python projects
License: MIT
Requires Python: >=3.10

# project-toolkit

A modular Python toolkit for managing project paths, cloud services (Google Cloud, Cloudflare R2), HashiCorp Vault, and configuration — powered by **pydantic-settings**.

## Features

- ⚙️ **Auto-Initialized Settings** — Just import and use, no setup needed
- 🔑 **Dynamic Env Access** — Read any env var or `.env` value with `settings.env("KEY")`
- 🔐 **Vault Integration** — Read secrets like `vault kv get` / `vault kv get -field=`
- 📁 **Project Path Management** — Create and navigate project directory structures
- ☁️ **Cloudflare R2** — Upload/download files with `boto3` and `requests` clients
- 📊 **Google Services** — BigQuery, Drive, and Sheets integrations
- 🌐 **WAN IP Tracker** — Detect and log public IP address changes

## Installation

```bash
# Core only
pip install project-toolkit

# With specific extras
pip install "project-toolkit[google]"
pip install "project-toolkit[cloudflare]"
pip install "project-toolkit[vault]"
pip install "project-toolkit[wan-ip]"

# Everything
pip install "project-toolkit[all]"
```

---

## API Usage Guide

### 1. Auto-Initialized Settings (No Setup Needed)

Settings auto-initialize on first use — no boilerplate required:

```python
from project_toolkit import settings

# Access any module setting directly
print(settings.cloudflare.r2_access_token)
print(settings.google.google_service_account_json)
print(settings.vault.is_configured)
```

**Environment detection** is automatic:
- If `AIRFLOW_HOME` is set → loads `config/.env.airflow`
- Otherwise → loads `config/.env.dev`

The `config/` directory is located by searching from your **current working directory** first,
then walking up the directory tree. This means it works whether you run from the project root,
a subdirectory, or even a Jupyter notebook inside the project.

---

### 2. Dynamic Environment Variable Access

Read **any** env var or `.env` value — no need to pre-define it in code:

```python
from project_toolkit import settings

# Read API keys (from env vars or .env file)
openai_key = settings.env("OPENAI_API_KEY")
gemini_key = settings.env("GEMINI_API_KEY")
claude_key = settings.env("CLAUDE_API_KEY")

# With a default fallback
db_host = settings.env("DATABASE_HOST", "localhost")
debug = settings.env("DEBUG_MODE", "false")
```

**Priority**: Environment variable > `.env` file value > default

Your `config/.env.dev` can contain any key:

```env
OPENAI_API_KEY=sk-xxxxx
GEMINI_API_KEY=AIza-xxxxx
CLAUDE_API_KEY=sk-ant-xxxxx
DATABASE_HOST=db.example.com
```

---

### 3. HashiCorp Vault

Read secrets from Vault — same env vars as the `vault` CLI.

**Setup** — Add to `config/.env.dev`:

```env
VAULT_ADDR=http://192.168.12.2:8200
VAULT_TOKEN=hvs.your-vault-token
```

**Read all fields** (like `vault kv get secret/my-credentials`):

```python
from project_toolkit import settings

secret = settings.vault.read_secret("secret/my-credentials")
print(secret)
# {'username': 'admin', 'password': 'secret-password-123'}
```

**Read a single field** (like `vault kv get -field=OPENAI_API_KEY secret/api`):

```python
from project_toolkit import settings

# Single field reads
api_key = settings.vault.read_field("secret/api", "OPENAI_API_KEY")
password = settings.vault.read_field("secret/my-credentials", "password")

# With fallback default
db_pass = settings.vault.read_field("secret/db", "password", "default-pw")
```

**Use the hvac client directly** for advanced operations:

```python
from project_toolkit import settings

client = settings.vault.get_vault_client()
if client:
    # Write a secret
    client.secrets.kv.v2.create_or_update_secret(
        path="my-new-secret",
        secret={"api_key": "abc123"},
    )

    # List secrets
    secrets = client.secrets.kv.v2.list_secrets(path="", mount_point="secret")
    print(secrets["data"]["keys"])
```

---

### 4. Explicit `.env` File Path (Fallback)

If auto-detection cannot find your `config/.env.dev` — for example, when
running from a **Jupyter notebook**, a different working directory, or a
remote environment — you can pass the path explicitly:

```python
from project_toolkit import settings

# Point to your .env file explicitly
settings.configure(env_file="/path/to/your/config/.env.dev")

# Now use settings as normal
print(settings.cloudflare.r2_access_token)
print(settings.env("OPENAI_API_KEY"))
```

Or via `get_settings()` directly:

```python
from project_toolkit import get_settings

get_settings.cache_clear()
settings = get_settings(env_file="/path/to/your/config/.env.dev")
```

---

### 5. Manual Settings Override

For testing or custom configurations, construct `AppSettings` directly:

```python
from project_toolkit.settings.base import AppSettings
from project_toolkit.settings.cloudflare import CloudflareSettings
from project_toolkit.settings.google import GoogleSettings
from project_toolkit.settings.path import PathSettings
from project_toolkit.settings.vault import VaultSettings
from project_toolkit.settings.wan_ip import WanIpSettings

custom_settings = AppSettings(
    path_config=PathSettings(
        data_dir="/custom/data",
        project_name="my-project",
    ),
    cloudflare=CloudflareSettings(
        r2_access_token="custom-token",
        r2_account_id="custom-id",
    ),
    google=GoogleSettings(),
    vault=VaultSettings(),
    wan_ip=WanIpSettings(),
)

# Use the custom settings
print(custom_settings.cloudflare.r2_access_token)  # "custom-token"
print(custom_settings.env("R2_ACCESS_TOKEN"))       # reads from env/dotenv
```

To reset the cached auto-initialized singleton:

```python
from project_toolkit import settings, get_settings

# Reset cached settings (e.g., after changing env vars)
get_settings.cache_clear()
settings._reset()

# Next access will re-initialize
print(settings.cloudflare.r2_access_token)
```

---

### 6. Project Path Management

Manage project directory structures with automatic creation:

```python
from project_toolkit import DataPathConfig

# Direct usage
dpc = DataPathConfig(
    project_name="my-project",
    subproject="etl",
    data_dir="/data",
)
print(dpc.data_dir())        # /data
print(dpc.project_dir())     # /data/my-project
print(dpc.sub_project_dir()) # /data/my-project/etl

# Get a timestamped file path
filepath = dpc.get_filepath("output", "csv")
# /data/my-project/etl/output_20260209.csv
```

**From auto-initialized settings** (recommended):

```python
from project_toolkit import settings, DataPathConfig

dpc = DataPathConfig.from_settings(
    settings.path_config,
    project_name="my-project",
)
```

---

### 7. Cloudflare R2

```python
from project_toolkit import settings
from project_toolkit.cloudflare.boto3_client import CloudflareBoto3Client

# From auto-initialized settings
client = CloudflareBoto3Client.from_settings(settings.cloudflare)

# Upload a file
client.upload_file("my-bucket", "file.csv", open("/path/to/file.csv", "rb").read())

# List buckets
buckets = client.list_buckets()
```

---

### 8. Google Sheets

```python
from project_toolkit import settings
from project_toolkit.google.sheet import GoogleSheetsManager

sheets = GoogleSheetsManager.from_settings(settings.google)

# Read a sheet as a DataFrame
df = sheets.sheet_df(sheet_id="your-sheet-id")
print(df.head())
```

---

### 9. Google BigQuery

```python
from project_toolkit import settings
from project_toolkit.google.bigquery import BigQueryManager

bq = BigQueryManager.from_settings(settings.google)

# Run a query
df = bq.query_table("SELECT * FROM my_dataset.my_table LIMIT 10")
```

---

### 10. Google Drive

```python
from project_toolkit import settings
from project_toolkit.google.drive import GoogleDriveManager

drive = GoogleDriveManager.from_settings(settings.google)

# List files in a folder
files = drive.list_files(folder_id="your-folder-id")
```

---

### 11. WAN IP Tracker

```python
from project_toolkit.wan_ip import WanIpTracker

tracker = WanIpTracker(log_dir="/tmp/ip-logs")

# Get current IPs (includes datetime)
print(tracker)
# Date: 2026-02-09 22:31:45
# IPv4: 1.2.3.4
# IPv6: 2001:db8::1

# Log only if changed
entry = tracker.log_if_changed()
```

---

## Configuration

### Environment Variables

| Variable | Module | Description |
|---|---|---|
| `R2_ACCESS_TOKEN` | Cloudflare | Cloudflare R2 API token |
| `R2_ACCOUNT_ID` | Cloudflare | Cloudflare account ID |
| `R2_DOMAIN` | Cloudflare | R2 custom domain URL |
| `GOOGLE_SERVICE_ACCOUNT_JSON` | Google | Path to service account JSON |
| `TEST_GOOGLE_SHEET_ID` | Google | Test sheet ID (dev only) |
| `VAULT_ADDR` | Vault | Vault server address (e.g., `http://192.168.12.2:8200`) |
| `VAULT_TOKEN` | Vault | Vault authentication token |
| `WAN_IP_LOG_DIR` | WAN IP | Directory for IP change logs |
| `AIRFLOW_HOME` | Core | Auto-detected for env switching |

> **Any key** in your `.env` file or environment can be read dynamically
> via `settings.env("KEY")` — no need to define it in Python code.

### `.env` Files

Place config files in the `config/` directory:

```
config/
├── .env.dev        # Local development (loaded when no AIRFLOW_HOME)
└── .env.airflow    # Airflow production (loaded when AIRFLOW_HOME is set)
```

### `.env` File Detection Order

The toolkit searches for `config/` in this order:

1. `<cwd>/config/` — current working directory
2. Walk up from `cwd` (up to 5 parent levels)
3. Walk up from the package install location (works for editable installs)

If auto-detection fails, use `settings.configure(env_file=...)` to specify
the path explicitly.

### Priority Order

Environment variables > `.env` files > Vault secrets > defaults

### Troubleshooting: `.env` File Not Found

If settings load with `None` values, the `config/.env.dev` file was not
found. This commonly happens when:

- Running from a **Jupyter notebook** whose working directory differs from
  the project root.
- Running a script from a **different directory** than the project root.
- The package was installed via `pip install` (not editable mode) and
  there's no `config/` folder in the current working directory.

**Fix**: Call `configure()` before accessing settings:

```python
from project_toolkit import settings

settings.configure(env_file="/absolute/path/to/config/.env.dev")
print(settings.env("OPENAI_API_KEY"))  # ✓ works
```

---

## Development

```bash
# Install with all dependencies
pip install -e ".[all]"

# Run tests
pytest

# Run specific tests
pytest tests/test_settings.py -s -vv
pytest tests/test_vault.py -s -vv
pytest tests/test_auto_settings.py -s -vv
```

## Project Structure

```
project_toolkit/
├── __init__.py           # Package exports + auto-initialized settings singleton
├── project_path.py       # Project path management (DataPathConfig)
├── settings/
│   ├── __init__.py       # Settings exports
│   ├── base.py           # AppSettings + get_settings() + env()
│   ├── path.py           # PathSettings
│   ├── cloudflare.py     # CloudflareSettings
│   ├── google.py         # GoogleSettings
│   ├── vault.py          # VaultSettings + read_secret() + read_field()
│   └── wan_ip.py         # WanIpSettings
├── cloudflare/
│   ├── boto3_client.py   # R2 via boto3
│   └── requests_client.py # R2 via requests
├── google/
│   ├── bigquery.py       # BigQuery operations
│   ├── drive.py          # Google Drive operations
│   └── sheet.py          # Google Sheets operations
└── wan_ip/
    └── tracker.py        # WAN IP detection + logging
```

## License

MIT