PSLD/PSLD · Apply for community grant: Academic project (gpu)

We present the first framework to solve general inverse problems leveraging pre-trained latent diffusion models. Previously proposed algorithms (such as DPS and DDRM) only apply to pixel-space diffusion models. We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.

In this web application, we demonstrate image inpainting using a pretrained Stable Diffusion generative foundation model. In the left panel, an image is first uploaded and then the eraser is used to mask out any content from this image. By clicking the "Inpaint" button, our posterior sampling algorithm PSLD is run in the background to generate missing parts that are consistent with the known portions of the image. The output samples are shown in the right panel.

We are currently getting OOM error, so we would like to request a grant for a GPU with higher memory.