[Official] RotEx: A Rotational Expert Network for Realistic Scene Illumination Variation Synthesis

📄 Paper: coming soon   |    💻 Source Code   |    🤗 Hugging Face

Introduction

RotEx is a transformer-based image-to-image framework for synthesizing scenes under varying illumination conditions. It combines content latents, depth information, style features, and a continuous time-of-day variable to generate illumination variations of an input scene.

Key Features

• Continuous illumination control: Generate scene variations across different times of the day.
• Illumination-dependent experts: Adapt style and image features through time-conditioned expert routing.
• Content and depth conditioning: Guide synthesis using content latents and estimated scene depth.
• Multi-step and one-step inference: Support full-model sampling and distilled single-step generation.
• Image and video processing: Process image directories and videos with reference-style conditioning.

Demo

Illumination Variation

Downstream Trajectory Comparison

Installation

Requirements: Python ≥ 3.10, PyTorch ≥ 2.6, a compatible torchvision version, FlashAttention ≥ 2.0, and an NVIDIA GPU with CUDA.

git clone https://github.com/val-utehy/RotEx.git
cd RotEx

conda create -n rotex python=3.10 -y
conda activate rotex
python -m pip install --upgrade pip

Install CUDA-enabled PyTorch and torchvision following the PyTorch instructions, then run:

python -m pip install packaging ninja psutil
python -m pip install "flash-attn>=2.0.0" --no-build-isolation
python -m pip install -r requirements.txt

Pretrained Models

Download weights from Hugging Face:

Model	Download	Save to
Multi-step	best.pt	checkpoints/full/best.pt
Multi-step, final	last.pt	checkpoints/full/last.pt
One-step	best_distill.pt	distill_checkpoints/best_distill.pt

hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
hf download val-utehy/RotEx distill_checkpoints/best_distill.pt --local-dir .
hf download black-forest-labs/FLUX.1-dev ae.safetensors --local-dir checkpoints

DINOv2 and Depth Anything V2 are downloaded automatically. Run hf auth login if model access requires authentication.

Expected Weights Structure

RotEx/
├── checkpoints/
│   ├── ae.safetensors
│   └── full/
│       ├── best.pt
│       └── last.pt                 # Optional
└── distill_checkpoints/
    └── best_distill.pt

Usage

1. Training

Will be updated soon.

2. Multi-Step Image Inference

python infer_multistep.py \
    --checkpoint checkpoints/full/best.pt \
    --content_dir inputs/content \
    --style_dir inputs/style \
    --output_dir outputs/multistep \
    --denoise_steps 50

3. One-Step Image Inference

Set these paths in infer_1step_style.py:

CHECKPOINT = "distill_checkpoints/best_distill.pt"
FIXED_STYLE_PATH = "inputs/style/night_reference.jpg"
DATA_PAIRS = [
    {"input": "inputs/content", "output": "outputs/1step_style"},
]
HOURS = list(range(1, 25))

python infer_1step_style.py

4. Multi-Step Video Inference

python infer_multistep_vid.py \
    --checkpoint checkpoints/full/best.pt \
    --video_path inputs/video.mp4 \
    --style_path inputs/style/night_reference.jpg \
    --output_dir outputs/video \
    --denoise_steps 50 \
    --hour_start 0 \
    --hour_end 24 \
    --no_audio

Citation

References

• FLUX
• DINOv2
• Depth Anything V2
• FlashAttention

License

This project is licensed under the Apache License 2.0. Third-party components retain their respective licenses.