Paper: PanoDiffusion: 360-degree Panorama Outpainting via Diffusion (ICLR-2024)

Reviewer comment (Experiments):
The proposed RGB-D framework is constructed by using two parallel LDM to reconstruct the depth and RGB images separately. This structure might result in a larger and more complex model architecture than using a shared or depth-conditional LDM.

Author response:
> Q1: The proposed RGB-D framework is constructed by using two parallel LDM to reconstruct the depth and RGB images separately. This structure might result in a larger and more complex model architecture than using a shared or depth-conditional LDM.

We do not use two parallel LDM's. As is claimed in Section 3.3, using two parallel LDMs to reconstruct Depth and RGB images separately will incur excessive computational resources. To avoid this, **our PanoDiffusion uses two VAE encoders, but only one LDM, with depth info as concatenated channels**. Therefore, the computational cost does not increase significantly compared to the original conditional LDM (see Q6).

Reviewer comment (Evaluation):
Apart from the visualization results, how about the evaluation results of using concatenation or cross-attention in the so-called depth-conditional diffusion model?

Author response:
> Q2: Apart from the visualization results, how about the evaluation results of using concatenation or cross-attention in the so-called depth-conditional diffusion model?

We've additionally tested depth-conditional diffusion models that use the usual concatenation and cross-attention conditioning on all types of masks. The results show our PanoDiffusion consistently outperforming the traditional depth-conditional LDM formulation.

|                          | Camera    |           |           |           | NFoV       |           |           |           |
| ------------------------ | --------- | --------- | --------- | --------- | ---------- | --------- | --------- | --------- |
| Method                   | FID       | sFID      | D         | C         | FID        | sFID      | D         | C         |
| PanoDiffusion            | **21.55** | **26.95** | **0.867** | **0.708** | **21.41**  | **27.80** | **0.790** | **0.669** |
| DC-LDM (cross-attention) | 51.77     | 37.96     | 0.275     | 0.280     | 48.14      | 39.17     | 0.248     | 0.271     |
| DC-LDM (concatenation)   | 114.59    | 51.96     | 0.023     | 0.018     | 112.54     | 64.80     | 0.046     | 0.006     |
|                          | **Layout**   |           |           |           | **Random Box** |           |           |           |
| Method                   | FID       | sFID      | D         | C         | FID        | sFID      | D         | C         |
| PanoDiffusion            | **23.06** | **22.39** | **1.000** | **0.737** | **16.13**  | **20.39** | **1.000** | **0.883** |
| DC-LDM (cross-attention) | 40.63     | 25.52     | 0.476     | 0.450     | 30.25      | 23.65     | 0.597     | 0.719     |
| DC-LDM (concatenation)   | 105.79    | 32.90     | 0.060     | 0.054     | 64.49      | 30.38     | 0.237     | 0.471     |

Reviewer comment (Experiments):
How is the cross-attention operation used in the depth-conditional method? Whether the authors try other advanced multimodal fusion methods to better combine the features from RGB and depth?

Author response:
> Q3: How is the cross-attention operation used in the depth-conditional method? Whether the authors try other advanced multimodal fusion methods to better combine the features from RGB and depth?

For depth-conditional LDM's, we pretrained a VQVAE for depth, and kept the encoder. The encoded depth is fed into the LDM as a condition in the usual two options, either via concatenation, or by cross-attention. Additionally, to improve robustness, we allowed the LDM to update the depth-condition encoder weights during training. However, these methods led to unsatisfactory outpainting results, often resulting in blurry content.

We did not investigate more advanced multimodal fusion baselines, because our main task is RGB **outpainting without any depth input at inference time**.

Reviewer comment (Experiments):
In the depth panoramic synthesis, how about the comparison to some RGB-based depth estimation methods? For example, in the case of using fully masked input.

Author response:
> Q5: In the depth panoramic synthesis, how about the comparison to some RGB-based depth estimation methods?

As suggested, we compared our PanoDiffusion with two RGB-based depth estimation methods, PanoFormer (ECCV'2022) [1] and ACDNet(AAAI'2022) [2], which are designed not only for depth estimation but specifically for panorama depth estimation. 

| **Methods**   | **RMSE ↓** | **MAE ↓** | **AbsREL ↓** | **Delta1.25 ↑** |
| ------------- | ---------- | --------- | ------------ | --------------- |
| ACDNet        | 353        | 209       | 0.2065       | 0.7804          |
| PanoFormer    | **268**    | **164**   | 0.2777       | 0.8863          |
| PanoDiffusion | 276        | 193       | **0.1355**   | **0.9060**      |

PanoDiffusion performed better than Panoformer on AbsREL and Delta1.25, but poorer on RMSE and MAE. We believe that this is likely due to the primary objectives of each model. For PanoDiffusion, we are more concerned with the outpainting quality of the RGB panoramas, with depth as auxiliary information. **Accurate depth synthesis reflects that PanoDiffusion has adequately learned the information about depth as expected, while more importantly having the ability to conduct high quality RGB outpainting.** Conversely, ACDNet and PanoFormer are designed to prioritize the quality of depth output, with RGB images serving as the primary input.

[1] Zhuang C, Lu Z, Wang Y, et al. ACDNet: Adaptively combined dilated convolution for monocular panorama depth estimation. Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(3): 3653-3661.

[2] Shen Z, Lin C, Liao K, et al. PanoFormer: Panorama Transformer for Indoor 360∘ Depth Estimation. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 195-211.

Reviewer comment (Evaluation):
How about the runtime or complexity analysis of the proposed method, since the parallel LDM are used for RGB and depth separately.

Author response:
> Q6: How about the runtime or complexity analysis of the proposed method, since the parallel LDM are used for RGB and depth separately.

As mentioned above, we do *not* use two parallel LDM's. Nonetheless, we conducted the runtime analysis as requested (also by reviewer MEmh). For training, we compared the average training time (minutes) for training one epoch, on the same hardware, batch size (4) and training dataset. For inference, we compared the time (seconds) required to infer a single image.

| Method        | Type                | Depth | Training (mins/epoch) | Inference (sec/image) |
| ------------- | ------------------- | ----- | --------------------- | --------------------- |
| PanoDiffusion | Bi-modal LDM        | +     | 82                    | 5                     |
| BIPS          | GAN                 | +     | 131                   | <1                    |
| RePaint       | Diffusion model     | -     | 78                    | 45                    |
| LDM           | LDM                 | -     | 72                    | 4                     |
| OmniDreamer   | Transformer + VQGAN | -     | 158                   | 61                    |

The results show that while our model is not the fastest, it remains within a reasonable range. It's also noteworthy that, compared to the original LDM framework, **our bi-modal structure achieves a significant improvement in the quality of outpainting.** **This improvement comes without a proportionate increase in resource consumption** – we observed only a modest increase of 13.8% in training time and 25% in inference time.

Reviewer comment (Novelty):
There is not many noval changes to the LDM framework. The authors only add a pretrained depth map encoder as a guidance.

Author response:
The more obvious approach is to train an LDM with depth masks as condition, particularly during training. We avoided this, and instead chose to use a RePaint-based structure, because we want to free the model from *presuming mask distributions*. This is critical for our two-end alignment design, as masks will continually change due to rotation. As LDM or Repaint cannot individually handle this challenge properly, it provides ample motivation for us to fuse these two components, naturally combined with our novel rotational denoising. Furthermore, we advanced our model to a bi-modal structure, where depth maps prove to have significant auxiliary effect in generating well-structured results. We also consider that our discovery that the *asymmetric* condition, where having a training setting (use of depth and without masking) that is substantially *different* from the inference setting (exclusion of depth and with masking), still leads to improved performance, is an interesting novel finding.

Reviewer comment (Experiments):
There are only one visual result (Figure 6) in the paper, which is not convincing. The authors should add more visual results for comparison in the supplementary materials.

Author response:
Perhaps it was overlooked, but we had previously included more visual results for comparison in our **Appendix, previously Fig. 12, now 13**, including two groups of examples for each mask type. Additionally, we had also provided some examples where we project different panorama images to 3D scenes in our supplementary video.

Reviewer comment (Experiments):
There are only 4 types of maskes. Each type of mask covers different portion of the whole panaroma. I recommand the authors to add a chart that describes the performance changing along the percentage of the mask. According to my understanding, with the increase of the percentage of the mask, the performance will drop.

Author response:
We appreciate your insightful suggestion. We've conducted additional experiments with different percentages of box masks to test the performance of the model, from 10% to 90%. Specifically, **we generated box masks of different total areas in percentage**, and no depth input is provided during inference. As the results suggest, the performance of PanoDiffusion declines with the increase in the mask percentage. This trend aligns with our expectations, as larger masks inherently pose a greater challenge for panorama image outpainting.

| Mask Percentage | FID   | sFID  | Density | Coverage |
| --------------- | ----- | ----- | ------- | -------- |
| 10%             | 6.58  | 9.80  | 1.000   | 1.000    |
| 30%             | 8.99  | 15.43 | 1.000   | 0.999    |
| 50%             | 12.63 | 20.56 | 1.000   | 0.980    |
| 70%             | 16.88 | 24.18 | 0.901   | 0.843    |
| 90%             | 22.94 | 28.04 | 0.644   | 0.661    |

We also calculated the percentage of each type of mask used in the paper. The results show that the performance is generally consistent with the mask percentage. Except for the layout mask, where we muffle all areas except floors and ceilings, it is more challenging to precisely outpaint all the details. Consequently, even though the layout mask percentage is close to the random box mask, its FID is higher. 

| Mask Type       | Camera | NFoV  | Layout | Random Box |
| --------------- | ------ | ----- | ------ | ---------- |
| Mask Percentage | 80.6%  | 85.4% | 50.6%  | 50.5%      |

Reviewer comment (Experiments):
The mask are all continuous presented in this paper, I am curious when there are several separate masks in one panaroma, what will the performance become?

Author response:
In our method, the design of the random box mask inherently includes 1 to 5 randomly sized bounding boxes. Consequently, **there is a probability of these masks being discontinuous**. Nonetheless, to explicitly assess our model's performance here in a more deterministic manner, we first note that for the camera mask type, the visible portions are definitely non-contiguous. So to test non-contiguous masks, we flipped the camera mask - swapping the originally visible and invisible parts to simulate this situation. This type of mask is equivalent to randomly sampling several NFoV masks and making them invisible. The quantitative results show that the model is still able to perform outpainting properly.  More qualitative results are provided in the Appendix of our paper for your reference.

| Method        | FID  | sFID  | Density | Coverage |
| ------------- | ---- | ----- | ------- | -------- |
| PanoDiffusion | 9.38 | 12.56 | 1.000   | 0.993    |

Reviewer comment (Reproducibility):
During the training stage, does the random angle rotation only apply to horizontal direction?

Author response:
During the training stage, we restrict random angle rotation solely to the horizontal direction. As panorama images typically follow equirectangular projection, the distortion increases non-uniformly towards the top and bottom poles. **Introducing random angle rotation in the vertical direction would lead to substantial changes in the projection results, and require more complex preprocessing, which will increase training costs.** We have added two examples where the camera has a 90-degree vertical rotation in our Appendix 1.3 Fig.~10. Conversely, **distortion is uniform horizontally** - image manipulation here only involves horizontal cropping and splicing, without the need for reprojection.

In terms of training efficacy, without providing camera viewpoints as part of the input, **adding vertical rotations does not improve model performance**. Instead, it can complicate the learning of equirectangular projection patterns, reducing overall effectiveness. Since the main objective of PanoDiffusion is to generate coherent and realistic panorama images, we focus more on data augmentation in the horizontal direction - the spatial wrap-around consistency.

Reviewer comment (Experiments):
This paper claims the camera rotations as one of the contributions, please provide an ablation study on how the camera rotation actually works effectively.

Author response:
We have previously provided some ablation results on this. **Please see Fig. 8 and Table 3, where we compared PanoDiffusion with and without the use of rotations.**

To further shed light on the reviewer's question, we **additionally explored the effect of the rotation angle, including 180°, 90° (chosen for our final result) and 45°, on the outpainting results.** The results show that the wrap-around consistency is improved across all settings. Compared to 180°, 90° leads to better consistency. However, diminishing the angle further to 45° did not lead to additional improvements. We believe this is reasonable, as the model is expected to generate coherent content when the two ends are in contact for enough denoising steps. Therefore, smaller rotation angles and longer connections do not necessarily lead to more consistent results. 

| Methods \ Mask Type | Camera | NFoV   | Layout | Random Box | End    |
| ------------------- | ------ | ------ | ------ | ---------- | ------ |
| w/o rotation        | 125.82 | 128.33 | 128.10 | 128.19     | 132.69 |
| 180°                | 95.11  | 96.57  | 90.93  | 85.23      | 119.60 |
| 90°                 | 90.41  | 89.74  | 88.01  | 85.04      | 116.77 |
| 45°                 | 90.67  | 90.25  | 87.65  | 86.50      | 112.47 |