zaruta/freemorph-mvp

FreeMorph / IMPUS MVP

Single-page FastAPI app that wraps the FreeMorph diffusion pipeline alongside the IMPUS perceptually-uniform morphing method into a simple UI. Upload two square-ish images, optionally describe them, pick the backend, and receive the interpolated frames rendered by Stable Diffusion (2.1 for FreeMorph, 1.4+LoRA for IMPUS).

Heads up: Both pipelines are GPU hungry. FreeMorph prefers ≥12GB VRAM; IMPUS often needs 14GB+ plus long per-job fine-tuning runs.

Prerequisites

1. Python 3.10+ and CUDA-compatible PyTorch build (if you plan to use GPU).
2. Install dependencies (pin matches the FreeMorph paper setup):

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

The first run will download Stable Diffusion 2.1 weights as well as tokenizer / text encoder checkpoints into the local Hugging Face cache (~/.cache/huggingface).

Running the server (local GPU/CPU)

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Then open http://localhost:8000 and:

1. Upload two images (.jpg, .png, etc.). Square crops work best; non-square inputs are center-cropped internally.
2. (Optional) Provide short comma-separated prompts describing each photo.
3. Choose a morphing backend:
   • FreeMorph (SD 2.1) – faster baseline, matches the original paper settings.
   • IMPUS (SD 1.4, ICLR 2024) – perceptually-uniform sampling via textual inversion + LoRA fine-tuning per job. Expect much longer runtimes (tens of minutes) and higher VRAM usage (14GB+ strongly recommended).
4. Click Morph Images and wait for the frames to be generated.

Results are written to storage/results/<job_id> and exposed via /storage/results/... URLs so they can be re-loaded or downloaded later. Original uploads are kept under storage/uploads/<job_id> for reproducibility.

Deploying to Hugging Face Inference Endpoints

1. Login & create repo

   huggingface-cli login --token <your_hf_token>
   huggingface-cli repo create zaruta/freemorph-mvp --type=model
   git clone https://huggingface.co/zaruta/freemorph-mvp
   cd freemorph-mvp
   cp -R /Users/zaruta/morph-test/* .
   git add .
   git commit -m "Deploy FreeMorph"
   git push
   

2. Build & push Docker image (linux/amd64)

   cd /Users/zaruta/morph-test
   docker login -u zaruta
   docker buildx build --platform linux/amd64 -t zaruta/freemorph-mvp:latest --push .
   

3. Create endpoint
   In https://ui.endpoints.huggingface.co/ → New Endpoint:

   • Repository: zaruta/freemorph-mvp
   • Inference Engine: Custom
   • Container URL: docker.io/zaruta/freemorph-mvp:latest
   • Container port: 8000
   • Health route: /api/health
   • Hardware: GPU Medium (A10G), Region: us-east-1

   After the build finishes the endpoint URL looks like https://nziq03no5bepo0w1.us-east-1.aws.endpoints.huggingface.cloud.

Running the UI against the HF endpoint

cd /Users/zaruta/morph-test
source .venv/bin/activate
export API_BASE_URL="https://nziq03no5bepo0w1.us-east-1.aws.endpoints.huggingface.cloud"
export HF_API_TOKEN="<your_hf_token_here>"
uvicorn app.main:app --host 0.0.0.0 --port 8000

• The browser always talks to http://localhost:8000.
• FastAPI proxies /api/morph to the remote endpoint, attaching Authorization: Bearer <your_hf_token> when HF_API_TOKEN is set.
• Frames in the JSON response are rewritten to absolute URLs such as https://nziq03no5bepo0w1.us-east-1.aws.endpoints.huggingface.cloud/storage/results/<job>/frame_00.png, so the UI renders images directly from HF storage (локально они не появляются).
• The engine form field is forwarded to the remote endpoint, so hosted deployments expose the same FreeMorph/IMPUS toggle.

Switching between FreeMorph and IMPUS

• The UI dropdown updates the engine form field.
• Locally, FastAPI instantiates the requested service on first use and caches it for subsequent jobs.
• For engine=freemorph, diffusion runs fully in-process via FreeMorphService.
• For engine=impus, the app shells out to the vendored IMPUS reference script (vendor/IMPUS/run_morph.py), waits for it to dump numbered PNG frames, and renames them to the common frame_##.png format consumed by the UI.
• Both engines return the identical JSON contract (jobId, frames, frameCount, durationMs), so the UI and remote deployments stay agnostic.
• The IMPUS files are pulled verbatim from the official repo (MIT License) to stay close to the paper's implementation.

Tips & troubleshooting

• Generation is slow on CPU and may take many minutes per request. For a quick proof-of-concept, switch to a CUDA runtime.
• If VRAM is tight, lower steps, guidance_scale, or interpolation_size in app/freemorph_service.py.
• IMPUS performs textual inversion + LoRA finetuning per job. Expect 30–60 minutes on an A10G/A100 and keep /storage/results/<job> around if you want to inspect the intermediate .pt checkpoints it emits.
• The UI surfaces server errors directly. Inspect the FastAPI logs for PyTorch or diffusion-specific stack traces.

Project layout

• app/ – FastAPI entrypoint plus the FreeMorph/IMPUS service wrappers
• static/ & templates/ – MVP UI assets
• storage/uploads, storage/results – persisted inputs/outputs
• vendor/FreeMorph – untouched upstream FreeMorph implementation
• vendor/IMPUS – upstream IMPUS script + helpers invoked via subprocess

Future ideas

• Async job queue to avoid blocking requests while diffusion runs
• Automatic captioning (e.g., BLIP or Llava) to remove manual prompt entry
• Animated GIF/video export for the generated frames

Deploying to Hugging Face Inference Endpoints

The repo already contains a production-ready Dockerfile. To run FreeMorph on a hosted GPU:

1. Create/choose an HF repo (private is fine) and push this project (Dockerfile must live in the root).
2. In the HF UI go to Endpoints → Create endpoint, pick Custom Docker, select GPU hardware (A10G works well), and point the endpoint at your repo.
3. The build system will run pip install -r requirements.txt and start FastAPI via uvicorn app.main:app --host 0.0.0.0 --port 8000.
4. When the endpoint status becomes Running, your API will be reachable at https://<endpoint>/api/morph; reuse the same JSON contract as locally.

Stop/pause the endpoint when you are not testing to avoid GPU charges.