sideloader-ng 3.1.4
Download large files via PyPI packages
Requires Python: >=3.12

# 🚀 Sideload

Download large files via PyPI or npm packages! Sideload automatically splits large files into package-compliant chunks and allows you to download them through a beautiful CLI interface.

## Features

- ✨ **Beautiful CLI** powered by Rich with progress bars and status updates
- 📦 **Automatic file splitting** into 95MB package-compliant chunks
- 🔄 **Automatic reassembly** of downloaded parts
- 🎯 **Dual backend support** - Use PyPI or npm as temporary storage
- 🌐 **JSONBin integration** for request tracking
- 🛡️ **Error handling** with detailed progress monitoring
- 🧹 **Easy cleanup** - npm packages can be unpublished (much easier than PyPI!)

## Installation

```bash
git clone <repository-url>
cd Sideload
uv install
```

## Usage

### Environment Variables

Set up your credentials:

```bash
# Required
export JSONBIN_TOKEN="your_jsonbin_token"
export SIDELOAD_COLLECTION_ID="your_collection_id"

# Choose storage backend (default: pypi)
export STORAGE_BACKEND="npm"  # or "pypi"

# For PyPI backend
export PYPI_TOKEN="your_pypi_token"
export PYPI_USER="your_pypi_username"
export PYPI_PASSWORD="your_pypi_password"
export PYPI_TOTP="your_totp_secret"  # Optional, for 2FA

# For npm backend
export NPM_REGISTRY="https://registry.npmjs.org/"  # Optional, defaults to npmjs.org
export NPM_BASE_PACKAGE="sideloader-base-pkg"      # Existing npm package used for all sideload payloads

# npm token regeneration (used when sideloader-npm-token cache is missing/expired)
export NPM_USER="your_npm_username"
export NPM_PASSWORD="your_npm_password"
export NPM_TOKEN_SECRET_NAME="sideloader-npm-token"  # Kubernetes secret cache name
export NPM_TOKEN_EXPIRATION_DAYS="90"
export NPM_PLAYWRIGHT_HEADLESS="false"               # Optional: run headed browser
export NPM_PLAYWRIGHT_USE_XVFB="true"                # Optional: use virtual framebuffer when headed
export NPM_XVFB_DISPLAY=":99"                        # Optional
export NPM_XVFB_SCREEN="1920x1080x24"                # Optional
export NPM_PLAYWRIGHT_DEBUG="true"                   # Optional: verbose Playwright logs
export NPM_PLAYWRIGHT_DEBUG_DIR="/tmp"               # Optional: screenshots on failures/challenges

# Mail OTP extraction (IMAP inbox checked by sideloader when npm asks OTP)
export NPM_OTP_EMAIL_IMAP_HOST="imap.your-provider.com"
export NPM_OTP_EMAIL_IMAP_PORT="993"
export NPM_OTP_EMAIL_USERNAME="bot-mailbox@example.com"
export NPM_OTP_EMAIL_PASSWORD="your_mailbox_password"
export NPM_OTP_EMAIL_FOLDER="INBOX"  # Optional
export NPM_OTP_EMAIL_FROM="support@npmjs.com"  # Optional
export NPM_OTP_EMAIL_SUBJECT="Your npm one-time password"  # Optional
```

When running the server Docker image, headed Playwright with `NPM_PLAYWRIGHT_USE_XVFB=true`
is automatically wrapped in `xvfb-run` by the container entrypoint.

> **npm token caching behavior**:
> - sideloader reads token cache from Kubernetes secret `sideloader-npm-token`
> - if the cached token is expired/near-expiry, sideloader regenerates a token via Playwright
> - OTP is read from email inbox (IMAP) and extracted automatically
> - regenerated token is stored in the secret with `created_at` and `expires_at` (+90 days by default)

### Storage Backend Comparison

| Feature | PyPI | npm |
|---------|------|-----|
| Package limit | 100 MB | Unlimited |
| Cleanup | Browser automation required | Simple CLI command |
| Unpublish | Complex, requires 2FA | Easy with `npm unpublish` |
| Token Management | Manual (varies) | Auto-regenerated + cached in Kubernetes secret |
| Speed | Fast | Fast |
| Recommended for | Single-use uploads | Frequent testing |

**npm is recommended** for development and testing due to its much simpler cleanup process!

### Download a File

```bash
# Basic usage
uv run sideload download https://example.com/largefile.zip

# Specify output directory
uv run sideload download https://example.com/largefile.zip --output ./downloads/

# Override credentials
uv run sideload download https://example.com/largefile.zip --token YOUR_TOKEN --collection YOUR_COLLECTION
```

### How it Works

1. **Submit Request**: The CLI creates a new request in your JSONBin collection
2. **Monitor Progress**: Real-time progress monitoring with beautiful progress bars
3. **Server Processing**: Server downloads the file and splits it into chunks
4. **Upload to Storage**: Chunks are uploaded to PyPI or npm (depending on `STORAGE_BACKEND`)
5. **Download Packages**: CLI automatically downloads all packages containing file parts
6. **Reassemble**: Extracts and concatenates parts to rebuild the original file
7. **Cleanup**: Use cleanup scripts to remove packages after download

### Cleanup

#### npm (Recommended - Much Easier!)

```bash
# Clean up specific packages
python src/sideloader/scripts/cleanup_npm.py sideloader-base-pkg@1.1700000000.123 sideloader-base-pkg@1.1700000000.124

# Or use the server's built-in cleanup
uv run sideloader-server --clean-request-id YOUR_BIN_ID
```

npm packages can be easily unpublished using the npm CLI, making cleanup much simpler than PyPI!

#### PyPI (Requires Browser Automation)

```bash
# Clean up specific packages
python src/sideloader/scripts/cleanup_pypi.py package_name1 package_name2

# Or use the server's built-in cleanup
uv run sideloader-server --clean-request-id YOUR_BIN_ID
```

PyPI cleanup uses Playwright browser automation since PyPI doesn't provide a simple unpublish API.

### CLI Interface

The CLI provides:

- 🌈 **Colorful output** with status indicators
- 📊 **Progress bars** for downloads and processing
- 📈 **Real-time monitoring** of server-side processing
- ✅ **Success/error reporting** with detailed information
- 📋 **Summary tables** showing download statistics

### Example Output

```
🚀 SIDELOAD
Download large files via PyPI/npm packages

🌐 Requesting download for: https://example.com/largefile.zip
✅ Created sideload request: abc123def456

📡 Monitoring Progress
📥 Downloading... (45%) ████████████████▌                 
🔨 Building packages...
📤 Uploading part 1/3...

📊 Download Summary
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property          ┃ Value                           ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Original Filename │ largefile.zip                   │
│ File Size         │ 250,123,456 bytes               │
│ Total Packages    │ 3                               │
│ Status            │ ✅ UPLOADED                     │
└───────────────────┴─────────────────────────────────┘

📦 Downloading Packages
📦 Downloading package 1/3... ████████████████████████████

🔧 Reassembling File
🔗 Assembling part 1/3... ████████████████████████████

✨ Complete
🎉 File successfully downloaded to: largefile.zip
📊 File size: 250,123,456 bytes
```

## Development

### Server Setup

The server component handles file processing and PyPI uploads:

```bash
# Set environment variables
export JSONBIN_TOKEN="your_token"
export PYPI_TOKEN="your_pypi_token"

# Run the server
uv run python src/sideloader/main.py
```

### Kubernetes Deployment

Before installing the Helm chart, create the required secret in your namespace:

```bash
kubectl create secret generic sideload-secrets \
  --namespace <namespace> \
  --from-literal=jsonbin-token="YOUR_JSONBIN_TOKEN" \
  --from-literal=pypi-token="YOUR_PYPI_TOKEN" \
  --from-literal=pypi-user="YOUR_PYPI_USERNAME" \
  --from-literal=pypi-password="YOUR_PYPI_PASSWORD" \
  --from-literal=pypi-totp="YOUR_PYPI_TOTP_SECRET" \
  --from-literal=npm-token="OPTIONAL_EXISTING_NPM_TOKEN" \
  --from-literal=npm-user="YOUR_NPM_USERNAME" \
  --from-literal=npm-password="YOUR_NPM_PASSWORD" \
  --from-literal=npm-otp-email-imap-host="imap.your-provider.com" \
  --from-literal=npm-otp-email-imap-port="993" \
  --from-literal=npm-otp-email-username="bot-mailbox@example.com" \
  --from-literal=npm-otp-email-password="your_mailbox_password" \
  --from-literal=npm-otp-email-folder="INBOX" \
  --from-literal=npm-otp-email-from="support@npmjs.com" \
  --from-literal=npm-otp-email-subject="Your npm one-time password"
```

`sideloader-npm-token` is managed automatically by sideloader and does not need manual creation.
If you want to create it yourself, `--dry-run-npm-token` now prints a full Secret YAML manifest.

### npm Token Dry-Run

Use this to validate cache read/regenerate/write behavior without processing a sideload request:

```bash
# Standard flow: use cached token if valid, otherwise regenerate/write cache
uv run sideloader-server --dry-run-npm-token

# Force regeneration flow (Playwright + mail OTP + cache write)
uv run sideloader-server --dry-run-npm-token --force-regenerate-npm-token
```

Docker-based dry-run test (with virtual framebuffer):

```bash
# Build test image
docker build -f Dockerfile.npm-dryrun -t sideloader-npm-dryrun .

# Run dry-run + force regeneration
docker run --rm -it \
  -e PYTHONUNBUFFERED=true \
  -e NPM_USER="your_npm_username" \
  -e NPM_PASSWORD="your_npm_password" \
  -e NPM_OTP_EMAIL_IMAP_HOST="imap.gmail.com" \
  -e NPM_OTP_EMAIL_IMAP_PORT="993" \
  -e NPM_OTP_EMAIL_USERNAME="your.address@gmail.com" \
  -e NPM_OTP_EMAIL_PASSWORD="your_16_char_google_app_password" \
  -e NPM_OTP_EMAIL_FROM="support@npmjs.com" \
  -e NPM_OTP_EMAIL_SUBJECT="Your npm one-time password" \
  sideloader-npm-dryrun \
  --dry-run-npm-token --force-regenerate-npm-token
```

Note: this local Docker run is not in-cluster Kubernetes, so it will print Secret YAML for manual apply instead of writing to Kubernetes automatically.

### npm Publish/Unpublish Test Commands

Use these to test npm publishing with an existing base package:

```bash
# Upload a local file as one npm artifact
uv run sideloader-server --npm-test-upload-file /absolute/path/to/file.zip

# Optional custom dist-tag
uv run sideloader-server --npm-test-upload-file /absolute/path/to/file.zip --npm-test-tag sideload-docker-compose-f249932

# Unpublish a specific uploaded version
uv run sideloader-server --npm-test-unpublish sideloader-base-pkg@1.1700000000.123456
```

Then install the chart:

```bash
helm install sideload ./helm --namespace <namespace>
```

For npm token regeneration with headed Playwright + Xvfb in Kubernetes, ensure Helm keeps:
- `serverRunAsRoot: true`
- `npm.playwrightHeadless: false`
- `npm.playwrightUseXvfb: true`

### Project Structure

```
src/sideloader/
├── __init__.py           # Package initialization
├── cli.py                # CLI client
├── server.py             # Server component
├── jsonbin_connector.py  # JSONBin API wrapper
├── pypi_cleanup.py       # PyPI package deletion via Playwright
└── scripts/
    └── cleanup_pypi.py   # Admin script to delete all sideload packages
```

## License

[Your License Here]