MatFuse: Controllable Material Generation with Diffusion Models

🧩 Model Overview

MatFuse leverages diffusion models to simplify the creation of Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) maps. It allows for fine-grained control over material synthesis through multiple conditioning sources like color palettes, sketches, text, and images. Additionally, it supports post-generation editing of materials.

For more details, visit the project page or read the full paper on arXiv.

🧑‍💻 Usage

💿 Installation

Clone the repository:

git clone https://github.com/giuvecchio/matfuse-sd.git
cd matfuse-sd

Set up the environment:

# create environment (can use venv instead of conda)
conda create -n matfuse python==3.10.13
conda activate matfuse
# install required packages
pip install -r requirements.txt

Download the checkpoint.

🧪 Inference

To run inference on a trained model:

python src/gradio_app.py --ckpt <path/to/checkpoint.ckpt> --config src/configs/diffusion/<config.yaml>

📜 Citation

@inproceedings{vecchio2024matfuse,
  author    = {Vecchio, Giuseppe and Sortino, Renato and Palazzo, Simone and Spampinato, Concetto},
  title     = {MatFuse: Controllable Material Generation with Diffusion Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {4429-4438}
}

License

This project is licensed under the MIT License.

gvecchio
/

MatFuse