MatFuse: Controllable Material Generation with Diffusion Models

🧩 Model Overview

MatFuse leverages diffusion models to simplify the creation of Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) maps. It allows for fine-grained control over material synthesis through multiple conditioning sources like color palettes, sketches, text, and images. Additionally, it supports post-generation editing of materials.

For more details, visit the project page or read the full paper on arXiv.

🧑‍💻 Usage

💿 Installation

1. Clone the repository:

   git clone https://github.com/giuvecchio/matfuse-sd.git
   cd matfuse-sd
   

2. Set up the environment:

   # create environment (can use venv instead of conda)
   conda create -n matfuse python==3.10.13
   conda activate matfuse
   # install required packages
   pip install -r requirements.txt
   

3. Download the checkpoint.

🧪 Inference

To run inference on a trained model:

python src/gradio_app.py --ckpt <path/to/checkpoint.ckpt> --config src/configs/diffusion/<config.yaml>

📜 Citation

@inproceedings{vecchio2024matfuse,
  author    = {Vecchio, Giuseppe and Sortino, Renato and Palazzo, Simone and Spampinato, Concetto},
  title     = {MatFuse: Controllable Material Generation with Diffusion Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {4429-4438}
}

License

This project is licensed under the MIT License.