azure-blob-tui 0.2.1
changeme
Requires Python: <4.0,>=3.11

# azure-blob-tui

Terminal TUI for browsing Azure Blob Storage, plus Python helpers to read/write blobs
directly from your training code (no local files required).

## First-time setup

Run the TUI once to configure account/container/prefix and (optionally) store SAS
in an encrypted local file. On later runs, if SAS is stored, you only need the
passphrase (no need to re-enter SAS).

```bash
azure-blob-tui --configure
```

During first run you will be prompted for:

- account name / container name / default prefix
- whether to store SAS in an encrypted local file
- (if yes) the SAS token and a passphrase to encrypt it

Later runs:

- If SAS is stored, you will only be prompted for the passphrase.
- If you set `AZURE_BLOB_TUI_PASSPHRASE`, no prompt is needed.

## Use the TUI

```bash
azure-blob-tui
```

## Reconfigure

```bash
azure-blob-tui --configure
```

## Python API (no local files)

All helpers use the same config (account/container/default prefix) and SAS token
stored by `azure-blob-tui --configure`. If `AZURE_BLOB_TUI_PASSPHRASE` is set,
no prompt is needed.

Default prefix behavior:

- Default prefix is **not** applied automatically.
- Pass `use_default_prefix=True` to opt in per call.

### blob_open (file-like stream)

```python
import torch
from azure_blob import blob_open

# Save directly to Blob
with blob_open("checkpoints/step-100/model.pt", "wb") as f:
    torch.save(model.state_dict(), f)

# Load directly from Blob
with blob_open("checkpoints/step-100/model.pt", "rb") as f:
    state = torch.load(f, weights_only=False)
```

### Save JSON/YAML/text to Blob

```python
import io
import json
from azure_blob import blob_open

with blob_open("artifacts/config.json", "wb") as raw:
    with io.TextIOWrapper(raw, encoding="utf-8") as f:
        json.dump({"lr": 1e-4}, f)
```

### blob_url (signed URL helper)

```python
from azure_blob import blob_url

url = blob_url("images/cat.png")
```

### Storage helpers

```python
from azure_blob import (
    clear_cache,
    download_dir,
    download_file,
    list_blobs,
    prefetch_blobs,
    read_blob_bytes,
    upload_dir,
    upload_file,
)

upload_file("local.txt", "artifacts/local.txt")
download_file("artifacts/local.txt", "local_copy.txt", backend="auto", use_cache=False)

# Preload to local cache (faster repeated reads)
prefetch_blobs(["artifacts/local.txt", "artifacts/another.txt"])

# Reused metadata: cache on demand
meta = read_blob_bytes("artifacts/index.jsonl", use_cache=True)

# Clear cached objects (single blob or whole prefix)
clear_cache(blob_name="artifacts/local.txt")
clear_cache(prefix="artifacts/")

for name in list_blobs(prefix="artifacts/"):
    print(name)
```

### BlobContext (high-level workflow helper)

`BlobContext` wraps a configured `ContainerClient` and adds convenience helpers
for prefix resolution, listing, and in-memory read/write.

```python
import os
from azure_blob import BlobContext

os.environ["AZURE_BLOB_TUI_PASSPHRASE"] = "your-passphrase"

ctx = BlobContext.from_config()

for prefix in ctx.iter_prefixes(prefix="overview/among/"):
    print(prefix)

data = ctx.read_bytes("overview/among/123/topdown.png")
# Cache only when repeated reads are expected (for example JSONL metadata)
meta = ctx.read_bytes("dataset/index.jsonl", use_cache=True)
# For strict freshness, bypass cache:
latest = ctx.read_bytes(
    "overview/among/123/topdown.png",
    backend="auto",
    force_refresh=True,
)
ctx.write_bytes(
    "overview/among/123/processed.png",
    data,
    content_type="image/png",
)
```

## Fast Read Mode

Read-heavy APIs support `backend`:

- `backend="auto"` (default): use AzCopy for larger/batch reads when available, otherwise SDK.
- `backend="sdk"`: force Azure Python SDK path.
- `backend="azcopy"`: force AzCopy path (requires AzCopy + SAS).

`backend="auto"` selects AzCopy only when AzCopy is installed and a SAS token is
available; otherwise it falls back to SDK automatically.

Optional environment variables:

- `AZURE_BLOB_TUI_READ_BACKEND=auto|sdk|azcopy`
- `AZURE_BLOB_TUI_READ_WORKERS=16` (directory and prefetch parallelism)
- `AZURE_BLOB_TUI_MAX_CONCURRENCY=8` (per-blob SDK transfer concurrency)
- `AZURE_BLOB_TUI_AZCOPY_PATH=/path/to/azcopy`

Install AzCopy (optional, recommended for faster bulk download):

```bash
azcopy --version
```

## Cache Behavior

Read APIs are non-cached by default (`use_cache=False`).

- Cache default location: `~/.cache/azure-blob-tui/blob-cache`
- Override with: `AZURE_BLOB_TUI_CACHE_DIR=/your/cache/dir`
- Enable globally with: `AZURE_BLOB_TUI_CACHE_ENABLED=1`
- Cache hard limit (default 10GB): `AZURE_BLOB_TUI_CACHE_MAX_BYTES=10GB`
- Force bypass cache per call with: `force_refresh=True`

If files are updated by other writers, either:

- call read APIs with `force_refresh=True`, or
- clear stale entries via `clear_cache(...)`.

When cache exceeds the configured hard limit, the oldest cached files are deleted
automatically.

`AZURE_BLOB_TUI_CACHE_MAX_BYTES` accepts raw bytes or unit suffixes (`KB`, `MB`, `GB`),
for example: `10737418240`, `10240MB`, `10GB`.

## Common Parameter Choices

- Realtime image fetching (use once): `download_file(..., use_cache=False, backend="auto")`
- Reused JSON/JSONL metadata: `read_blob_bytes(..., use_cache=True)` or `ctx.read_bytes(..., use_cache=True)`
- Strong freshness (ignore local cache): add `force_refresh=True`
- Batch folder pull: `download_dir(..., workers=16, backend="auto", use_cache=False)` for one-shot jobs

`download_file(local_path=...)` always writes the requested blob into your target
local path. `use_cache` only controls whether an extra cache copy is kept.

## Recommended Pattern (JSONL + Images)

```python
from azure_blob import read_blob_bytes, download_file
import json

# 1) Metadata is reused -> cache it
records = [
    json.loads(line)
    for line in read_blob_bytes("dataset/index.jsonl", use_cache=True)
    .decode("utf-8")
    .splitlines()
]

# 2) Images are one-shot -> do not cache
for rec in records:
    blob_name = rec["image_blob"]
    local_path = f"/tmp/images/{rec['id']}.jpg"
    download_file(blob_name, local_path, use_cache=False, backend="auto")
    # use image immediately ...
```

## Choosing the right helper

- `BlobContext`: best for workflows that need listing, grouping, and in-memory
    read/write without touching disk.
- `blob_open`: best for file-like streaming APIs (e.g., `torch.save/load`,
    `TextIOWrapper`, or very large files).
- `read_bytes`/`write_bytes`: best for small-to-medium blobs where a full in-memory
    buffer is fine (images, JSON, etc.).
- `list_blobs`/`iter_prefixes`: best for enumeration and directory-like traversal.
- `download_*`/`upload_*`: local file paths (avoid these if you want no local storage).
- `prefetch_blobs`: best for pre-warming cache before training/evaluation loops.

## Notes

- SAS tokens are not stored in the config file, but in an encrypted local file.
- `blob_open` works for any file type (torch checkpoints, JSON, YAML, text, etc.).