MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation

Here is a ML-generated summary

Objective
The paper proposes MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation.

Insights

1. Dividing the image into regions and generating multiple objects in one image is more effective than generating single objects.
2. Allowing a certain overlap between regions results in a smoother transition and better image quality.
3. Appending a category definition to the prompt reduces ambiguity and improves accuracy.
4. Aggregating attention maps across all layers and time steps captures both coarse shapes and fine details.
5. Higher quality diffusion models like Stable Diffusion produce better instance segmentation performance.
6. MosaicFusion consistently improves various detection architectures and is complementary to other data augmentation techniques.

Implementation

1. Divide the image canvas into multiple regions and define text prompts for each region specifying the object to generate.
2. Map the image canvas from pixel space to latent space based on the upscaling factor of the diffusion model.
3. Run the diffusion process on each latent region in parallel using a shared noise prediction model. Initialize with the same noise but condition each region on the corresponding text prompt.
4. Aggregate the cross-attention maps for each token across layers and time steps by upsampling and averaging.
5. Threshold the aggregated attention maps and refine the masks using edge-aware algorithms like bilateral solver.
6. Filter low-quality masks based on connected components and keep those with one component.
7. Expand the filtered region masks to the full canvas size to produce the final instance masks.

Results
MosaicFusion significantly boosts performance on long-tailed and open-vocabulary instance segmentation benchmarks, especially for rare and novel categories.