sip-videogen 0.4.4
Transform video ideas into complete videos using AI agent teams
License: MIT
Requires Python: >=3.11

# sip-videogen

CLI tool that transforms vague video ideas into complete videos using an AI agent team.

## How It Works

```
User Idea → AI Agent Script Team → Reference Images → Video Clips → Final Video
```

1. You provide a video idea (e.g., "A cat astronaut explores Mars")
2. AI agents collaborate to write a script with scenes and shared visual elements
3. Reference images are generated for visual consistency (characters, props, environments)
4. Video clips are generated for each scene using VEO 3.1 via Gemini API (8 seconds per clip)
5. Clips are assembled into a final video with background music via FFmpeg

## Installation

### Option 1: Install from PyPI (Recommended)

```bash
# Install pipx if you don't have it
pip install pipx

# Install sip-videogen
pipx install sip-videogen

# Run - first time will prompt for configuration
sipvid
```

On first run, you'll be prompted to paste your configuration. Just paste the entire config block and press Enter twice.

### Option 2: Run from Source

```bash
# Clone the repo
git clone https://github.com/chufeng-huang-sipaway/sip-videogen.git
cd sip-videogen

# Copy and fill in your API keys
cp .env.example .env

# Run (installs everything automatically on first run)
./start.sh
```

## Prerequisites

- Python 3.11+ (`brew install python@3.11` on macOS)
- FFmpeg (`brew install ffmpeg` on macOS)

### API Keys Required

Get these API keys:

| Key | Required | Where to get it |
|-----|----------|-----------------|
| `OPENAI_API_KEY` | Yes | [OpenAI Platform](https://platform.openai.com/api-keys) |
| `GEMINI_API_KEY` | Yes | [Google AI Studio](https://aistudio.google.com/apikey) |
| `GOOGLE_CLOUD_PROJECT` | Optional | [Google Cloud Console](https://console.cloud.google.com) - Only needed for Lyria background music |
| `KLING_ACCESS_KEY` | Optional | [Kling AI](https://app.klingai.com/global/dev/api-key) - Alternative video generator |
| `KLING_SECRET_KEY` | Optional | Required if using Kling |

## Configuration

### First-Time Setup

On first run, `sipvid` will prompt you to configure your environment. You can either:

1. **Paste a config block** (recommended) - Paste all your keys at once:
   ```
   OPENAI_API_KEY=sk-...
   GEMINI_API_KEY=AIza...
   ```

2. **Enter keys individually** - Follow the interactive prompts

Configuration is stored in `~/.sip-videogen/.env` and works from any directory.

### Managing Configuration

```bash
sipvid config          # Interactive config editor
sipvid config --show   # Show current configuration status
sipvid config --reset  # Replace config with a new config block
```

## Usage

### Interactive Menu

```bash
sipvid
```

This launches a simplified interactive menu:

```
? Use arrow keys to navigate, Enter to select:
❯ Generate Video     Create a new video from your idea
  View History       See previous generations
  More Options...    Settings, resume, and other tools
  Exit
```

### Director's Pitch Workflow

When you select "Generate Video", you'll experience a streamlined creative workflow:

1. **Enter your idea** - Describe your video concept
2. **Select duration** - Choose from 15s, 30s, 45s, or 60s
3. **Review the pitch** - The AI presents a "Director's Pitch" with:
   - Title and logline
   - Tone and visual style
   - Key elements and scene breakdown
4. **Approve or refine** - Accept the pitch, provide feedback for revision, or cancel
5. **Generate** - Once approved, full video generation begins

The AI agent team decides the optimal scene count based on your story's complexity.

### View History

Access all your previous video generations from the main menu:
- See title, date, duration, and completion status
- Resume or regenerate from any previous script
- Open output folders directly

### Direct Commands

```bash
# Generate a video (uses interactive pitch flow)
sipvid generate "A cat astronaut explores Mars"

# Regenerate videos from an existing run (reuse saved script + images)
sipvid resume output/sip_20251210_123855_e9a845e4

# Generate with specific number of scenes
sipvid generate "Epic space battle" --scenes 5

# Dry run (script only, no video generation)
sipvid generate "Underwater adventure" --dry-run

# Skip cost confirmation
sipvid generate "Robot dance party" --yes

# Check configuration status
sipvid status
```

### Brand Kit Generation

Generate a complete brand design library from a simple concept:

```bash
sipvid brandkit "A skincare product brand with a tropical, organic feel"
```

**How it works:**

1. **Describe your brand** - Provide a concept (product type, style, audience)
2. **Review the brief** - AI analyzes your concept and creates a brand brief
3. **Choose a direction** - Select from 3 creative directions with different palettes and styles
4. **Approve the logo** - Review and approve the generated logo before continuing
5. **Generate assets** - All other assets use your approved logo for visual consistency

**Logo approval with feedback loop:**

When the logo is generated, you can:
- **Click the file link** to open it directly from the terminal
- **Auto-open for preview** - Option to launch in your system's image viewer
- **Provide feedback** - If you don't like the logo, describe what to improve (colors, style, layout)
- **Iterate** - The AI regenerates the logo based on your feedback (up to 3 attempts)

**Generated assets include:**

| Category | Assets |
|----------|--------|
| Logo | Primary brand logo (used as reference for all other assets) |
| Packaging | Hero shot, alternate colorway |
| Lifestyle | In-use photo, flatlay, environment |
| Mascot | Primary and alternate poses |
| Marketing | Landing page, recipe card, merch, pop-up stand, meme |

The logo you approve becomes the visual anchor - it's passed as a reference image to Gemini when generating packaging and marketing assets, ensuring consistent branding across all outputs.

## Automatic Updates

The tool automatically checks for updates on each run. When a new version is available, you'll see a notification:

```
┌─────────────────────────────────────────────────┐
│  Update available!                              │
│  Current version: 0.1.0                         │
│  Latest version:  0.2.0                         │
│  Run: sipvid update                             │
└─────────────────────────────────────────────────┘
```

Update commands:

```bash
sipvid update         # Check and install updates
sipvid update --check # Only check, don't install
```

## Architecture

The tool uses a hub-and-spoke agent pattern:

- **Showrunner** (orchestrator) - Coordinates the script development process
  - **Screenwriter** - Creates scene breakdown with professional cinematography
  - **Production Designer** - Identifies shared visual elements for consistency
  - **Continuity Supervisor** - Validates consistency and optimizes prompts
  - **Music Director** - Designs complementary background music

### VEO 3.1 Prompt Optimization

Prompts are structured following [Google's VEO 3.1 best practices](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1):

- **Prompt stack (global → elements → shots)** keeps critical anchors short:
  - Global: tone + visual_style fragments (palette, light, camera vibe, treatment)
  - Entity cards: one line per shared element (role descriptor + 2–3 anchors)
  - Shots: timestamped micro-prompts `[00:00-00:04] 4s medium static on reporter asking question at snowy city curb; crowd 6-8 crosses behind; bright overcast; flow: no pauses`
- **Professional camera terminology**: dolly, tracking, crane shots with depth of field control
- **Dialogue integration**: Quotes and speaker attribution for natural delivery
- **Audio design**: `Ambient:` and `SFX:` prefixes for precise sound control
- **Continuity anchors**: per-shot “flow: keep motion continuous, no pauses” with explicit exits/entrances and persistent props/splatter when needed

This same structured stack is used for Kling prompts (trimmed to the 2.5k character limit) to preserve detail without overloading the model.

### Clip Duration & Clip Patterns

When using reference images for visual consistency (standard workflow), VEO generates **fixed 8-second clips**. To create rhythm and shot variety within this duration, the system uses **clip patterns** - pre-defined shot duration combinations that guarantee the math always works:

| Pattern | Name | Best For |
|---------|------|----------|
| `[8]` | Single continuous | Establishing shots, emotional moments |
| `[6, 2]` | Long + quick | Build-up with quick ending beat |
| `[2, 6]` | Quick + long | Hook attention, then develop |
| `[4, 4]` | Two equal | Action-reaction, call-response |
| `[4, 2, 2]` | Medium + two quick | Building intensity |
| `[2, 4, 2]` | Sandwich | Bookended emphasis |
| `[2, 2, 4]` | Two quick + medium | Quick start, sustained finish |
| `[2, 2, 2, 2]` | Four quick | Montage, high energy, rapid cuts |

The **Showrunner** selects patterns for each scene based on pacing needs, and the **Screenwriter** creates matching `sub_shots` using timestamp prompting:

```
[00:00-00:02] Wide establishing shot of the food truck
[00:02-00:04] Medium shot, the vendor prepares ingredients
[00:04-00:06] Close-up of sizzling grill
[00:06-00:08] Medium shot, vendor plates the food
```

This creates dynamic multi-shot sequences within a single clip, similar to professional editing - while ensuring shots never split across clips.

### Seamless Scene Flow

Video clips are generated in parallel for speed, but the system ensures smooth transitions:

- **Flow Context**: Each clip receives position-aware instructions (first/middle/last) to avoid awkward pauses
- **Scene Continuity**: Screenwriter creates scenes that flow seamlessly:
  - First scene: May open naturally, must end with action in progress
  - Middle scenes: Must begin AND end mid-action (no pauses)
  - Last scene: Must begin mid-action, may conclude naturally

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest

# Run specific test
python -m pytest tests/test_models.py -v

# Lint and format
ruff check .
ruff format .

# Type check
mypy src/
```

### Publishing New Versions

```bash
# 1. Bump version in pyproject.toml
# 2. Run publish script
./scripts/publish.sh
```

## Cost Estimation

Before generating videos, the tool displays estimated costs:
- Gemini image generation: ~$0.13-0.24 per image
- VEO video generation: Check current [Gemini API pricing](https://ai.google.dev/pricing)

Use `--yes` to skip the cost confirmation prompt.

## Video Providers

The tool supports multiple video generation providers:

| Provider | Description |
|----------|-------------|
| **VEO** (default) | Google's VEO 3.1 via Gemini API. Supports reference images for visual consistency. |
| **Kling** | Kling AI video generation. Faster but doesn't support reference images without GCS. |

Switch providers in the Settings menu or set a default in your preferences.