Paper: GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR-2024)

Reviewer comment (Experiments):
Lack of comparisons with the latest NeRF-based models, except Pixel-NeRF which is proposed in 2021.

Author response:
Thank you very much for your feedback. To address your concerns, we conducted additional experiments on RealEstate10K. 

>Lack of comparisons with the latest NeRF-based models, except Pixel-NeRF which is proposed in 2021. 

We added additional NeRF-based models from the Appendix of Du et al. 2023 into Table 3.
Additionally, we trained and evaluated the latest concurrent NeRF-based approach, MatchNeRF, from arXiv on RealEstate10k. We included these numbers in Table 3 in the revised manuscript. 

| Model                              	| PSNR  | LPIPS | SSIM   |
|----------------------------------------|:-------:|:-------:|:-------:|
| PixelNeRF (Yu et al. 2021)         	| 13.91 | 0.591 | 0.460 |
| StereoNeRF (Chibane et al. 2021)   	| 15.40 | 0.604 | 0.486 |
| IBRNet (Wang et al. 2021)          	| 15.99 | 0.532 | 0.484 |
| GeoNeRF (Johari et al. 2022)       	| 16.65 | 0.541 | 0.511 |
| MatchNeRF (Chen et al. 2023)       	| **23.06** | 0.258 | 0.830 |
| -------------------------------------- |-------|-------|-------|
| GPNR (Suhail et al. 2022)          	| 18.55 | 0.459 | 0.748 |
| (Du et al. 2023)                   	| 21.65 | 0.285 | 0.822 |
| (Du et al. 2023) + GTA (Ours)     | 22.85 | **0.255** | **0.850** |

We grouped the works into methods using NeRF-based rendering and methods using Transformer-based rendering. We find that MatchNeRF outperforms both NeRF-based and Transformer-based prior work by large margins. By combining our GTA mechanism with Du et al. 2023, the transformer-based method also achieves state-of-the-art results.
We would like to mention that the transformer-based NVS with GTA, which introduces a relatively weaker prior into the model, is competitive with the model equipped with a strong physical prior (volumetric rendering).  Our attention mechanism effectively steers the transformer to learn the underlying 3D information behind images.

Reviewer comment (Experiments):
The resolution of training images used in the experiment seems to be not high. How does this method perform on high-resolution data, and what is the impact of different numbers of training views?

Author response:
Thank you very much for your feedback. To address your concerns, we conducted additional experiments on RealEstate10K. 

>How does this method perform on high-resolution data, …

We conducted experiments on RealEstate10k with 384x384 resolutions (1.5 times larger in height and width than in Table 3). We see that GTA also improves over the baseline model at higher resolution. We added this table as Table 7.

|Method| PSNR  | LPIPS 	| SSIM |
|----|:-----------:|:-----------:|:-----------:|
| Du et al. 2023 | 21.29 | 0.341| 0.838 |
| Du et al. 2023 + GTA (ours)| **21.95** | **0.322** | **0.849** |

Due to limited computational resources and discussion period, we only trained for 100K iterations, which is one-third of the iterations in Table 3. We will include results with fully trained models in the final version.

>...what is the impact of different numbers of training views?

We trained models with 3-context views and show the results below: We see GTA improves reconstruction quality over Du et al. 2023 with 3 views setting. We added this table as Table 8.

| Method|  2-views PSNR | 3-views PSNR      |
|----------------|:-----------------:|:--------------:|
| (Du et al. 2023) |  	21.43  |            21.84      |
| (Du et al. 2023) + GTA (ours)     | 	**22.17** 	|         **22.50**    |

Also in this table, numbers are computed with models trained for 100K iterations, and we will include results with fully trained models in the final version.

Reviewer comment (Presentation):
This paper's organization could be improved. For instance, placing the related work (Section 5) after the introduction would help readers understand the difference between conventional positional encoding schemes and the proposed relative transformation encoding scheme. The authors should summarize existing geometric-related positional encoding techniques (not only in novel view synthesis, but also in many 3D vision tasks), and conduct a comprehensive comparison. Since the background information of transformer (the first half of section 2 Background) is already well-known, it can be moved to the appendix.

Author response:
Thank you very much for your feedback. In the revised version, we add a number of ablation studies and numerical comparisons to existing positional encoding schemes in 3D visions. We hope our new version of the manuscript and the response below will address your concerns.

>This paper's organization could be improved. For instance, placing the related work (Section 5) after the introduction would help readers understand the difference between conventional positional encoding schemes and the proposed relative transformation encoding scheme.

Thank you for your feedback and suggestions. As suggested, in the revised manuscript we put discussions of 3D position encodings in Section 2 and renamed it to Related work. 

> The authors should summarize existing geometric-related positional encoding techniques (not only in novel view synthesis, but also in many 3D vision tasks), and conduct a comprehensive comparison

In Related work, we added discussions of existing positional encoding (PE) schemes from other 3D vision communities besides NVS. Specifically, we include PEs used in state-of-the-art 3D object detection and segmentation tasks (Frustum embedding, Motion Layer normalization), PEs when depth supervision is available (3DPPE), and also we also include a recent PE method used in geometry-biased transformer (GBT), which biases the attention matrix of each layer by ray distance between each query and key pair. Also we added 3D positional embedding methods for point clouds. 

>Background info on TransFormer can be moved to the appendix.

Thank you for the suggestion. We make the transformer’s background more concise. We keep the introductions of APE and RPE in Section 2, since it should be helpful for readers to easily understand the difference to our proposed method.

Reviewer comment (Novelty):
The paper's contribution appears to be incremental, offering limited insights from the proposed relative transformation positional encoding alone.

Author response:
> The paper's contribution appears to be incremental, offering limited insights from the proposed relative transformation positional encoding alone.

As we describe in the related work section, most of the existing works of transformers use positional encoding methods that do not faithfully keep the geometric structure associated with tokens. We posit that the structure-preserving way to encode geometric information of tokens; i.e.) the homomorphism property of the transformations applied to QKV vectors is important. The homomorphic transformation makes features more structurally position-aware, and therefore, subsequent layers easily learn the effect of the positional encoding, which is supported by Figure 6, showing GTA significantly improves learning efficiency over SRT and RePAST.  
The element-wise multiplication experiment requested by reviewer ZuJ5 additionally supports this. In that experiment, we compare GTA against element-wise multiplication of features that encode geometric information. When the representations are transformed into vector form multiplied to QKV vectors, the homomorphism property is lost. Suppose $f(g) :=W vec(\rho(g))$ is the vector embedding of the representations $\rho(g)$, then $f(gg’) \neq f(g)f(g’)$ in general. Thus, f is not faithful in terms of structure preservation of $g$. We also see in the table below that the performance significantly degrades when using element-wise multiplication to the QKV features. 
| Method                  	|	PSNR 	|
|-----------------------------|:-----------:|
| Element-wise multiplication |  34.33  |
| GTA (ours)              	|  **38.99**  |

See also the response to reviewer myD9 regarding the importance of geometry-preserving positional encoding of GTA. 

RoPE (Su et al. 2021) is similar to GTA. In our manuscript, we broaden the scope of the transformations from just 1D sequence in RoPE to geometric structures such as SE(3) (camera transformations in our experiments) and can cover even more general geometric structures as long as they can be defined in terms of a group. We also form an intuitive explanation in Section 3. The transformation on V, which is missing in RoPE, is found to be important, and Table 4(c) shows removing transformations on V degrades performance significantly. This is expected from the intuition behind GTA. Value vectors, which will be added to other token features from different coordinate systems, should also be transformed with the relative transformation, so that the addition is performed in an aligned coordinate space. 

References
* Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multiview 3d object detection. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
* Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. arXiv.org, 2023.
* Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry-biased transformers for novel view synthesis. arXiv.org, 2023.
* Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv.org, 2021.
* Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Interpretable transformations with encoder-decoder networks. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.

Reviewer comment (Experiments):
While the paper introduces GTA as a novel positional encoding scheme, it seems to function more like a transformer block (consisting of positional encoding and the attention block) than just a positional encoding scheme. A more detailed ablation study of each component within GTA is needed.

Author response:
Thank you very much for your feedback. We plan to conduct additional experiments in response to your requests.

Regarding Weakness 3, where it is mentioned:
>A more detailed ablation study of each component within GTA is needed.

Could you please clarify what specific ablation studies you are envisioning? In Table 5, we have already demonstrated the results when removing the representation matrix applied to the value vector in Equation (4). Additionally, Table 4 shows the results using different representation matrices. Could you suggest what other types of ablation experiments you think would be beneficial?

Reviewer comment (Novelty):
Geometry-aware attention for transformers has been extensively explored, and the idea of encoding position and transformation information to improve 3D vision tasks has been discussed previously. While the detailed implementation of the encoding process (such as applying transformations to value vectors) differs from existing works, they share a similar underlying concept, thereby diminishing the novelty of this paper.

Author response:
>While the detailed implementation of the encoding process (such as applying transformations to value vectors) differs from existing works, they share a similar underlying concept

In our manuscript, we define our GTA in the language of group theory, which enables us to broaden the scope of the transformations from just 1D sequence in existing work (RoPE) to geometric structures such as SE(3) (camera transformations in our experiments) and can cover even more general geometric structures as long as they can be defined in terms of a group.  
Also, the transformation on V, which is missing in RoPE, is not a heuristic in GTA, but naturally arises in our formulation with an intuitive explanation. Value vectors, which will be added to other token features from different coordinate systems, should also be transformed with the relative transformation so that the addition is performed in an aligned coordinate space. Table 4(c) shows removing transformations on V degrades performance significantly.

Reviewer comment (Experiments):
The evaluation and ablation study primarily focus on the novel view synthesis task. However, since the proposed method is easily integrable into existing attention-based 3D vision tasks, I highly recommend validating the approach on other 3D vision tasks, such as multi-view 3D detection.

Author response:
>Since the proposed method is easily integrated into existing attention-based 3D vision tasks, I highly recommend validating the approach on other 3D vision tasks, such as multi-view 3D detection.
>Additionally, since geometry-aware attention is a general method, can it also enhance accuracy in other perceptual tasks apart from novel-view synthesis?

Indeed, the general idea of GTA is applicable to other 3D vision tasks. 
In 3D detection and segmentation, the target coordinate system is typically an orthographic birds eye view (BEV) grid which requires a different $\rho$ for the decoder. Additionally, methods in these communities work with high-resolution feature grids, requiring sparse attention mechanisms like deformable attention to work efficiently. We think extending GTA to sparse attention and designing a novel representation $\rho$ for orthographic target cameras deserves a more thorough investigation than is possible in the rebuttal period and is a promising direction for future work.

Reviewer comment (Theory):
Unlike the feature warping approach, the encoded transformation information relies on matrix multiplication. To be honest, I don't fully grasp the physical meaning behind this.

Author response:
> Unlike the feature warping approach, the encoded transformation information relies on matrix multiplication. To be honest, I don't fully grasp the physical meaning behind this.

We group the channels into multiple 3D points, which encourages the network to learn geometric features.  The attention layers are optimized to match the relevant query and key vectors, therefore each 3-dimensional feature is expected to behave as abstract point clouds. This type of composition of vector spaces is often used in the literature on equivariant neural networks (See Section3 of Deng et al. (2021) or Section 2.6 of Cohen and Welling (2017)) and in the literature on transforming autoencoders (Hinton et al. (2011), Worrall et al . (2017)).

Reviewer comment (Experiments):
Additionally, since geometry-aware attention is a general method, can it also enhance accuracy in other perceptual tasks apart from novel-view synthesis?

Author response:
>Additionally, since geometry-aware attention is a general method, can it also enhance accuracy in other perceptual tasks apart from novel-view synthesis?

Indeed, the general idea of GTA is applicable to other 3D vision tasks. 
In 3D detection and segmentation, the target coordinate system is typically an orthographic birds eye view (BEV) grid which requires a different $\rho$ for the decoder. Additionally, methods in these communities work with high-resolution feature grids, requiring sparse attention mechanisms like deformable attention to work efficiently. We think extending GTA to sparse attention and designing a novel representation $\rho$ for orthographic target cameras deserves a more thorough investigation than is possible in the rebuttal period and is a promising direction for future work.

Reviewer comment (Theory):
While I agree with the intuition behind this method, what the actual implementation does, and how it affects attention, is somewhat unclear to me. It makes sense to transform features of different views to canonical reference space during attention, and that's obvious only when those are actual coordinates (e.g. points in 3D space); it's not so obvious when features are linear projections of other features.

Author response:
We treated the channels into multiple 3D points, which encourages the network to learn geometric features. This composition of vector spaces is often used in the literature on equivariant neural networks (See Section3 of Deng et al. (2021) or Section 2.6 of Cohen and Welling (2017)) and in the literature on transforming autoencoders (Hinton et al. (2011), Worrall et al. (2017)). 

We would like to mention that one potential advantage of integrating geometric information as group representations lies in the structured nature of these representations, i.e. the homomorphism property of the representations. In our approach, each QKV vector is transformed by ρ(g) in the same way irrespective of its value, ensuring consistency of the transformed vectors. This makes it easier for the subsequent layers to understand the effect of the transformations. Conversely, with absolute positional encoding (APE) and relative positional encoding (RPE), where the flattened camera transformations are added directly to the input, challenges arise. Not only does the flattened matrix lose its geometric structure when merged with each $Q$, $K$, and $V$, but subsequent layers are also burdened with learning the effect of complicated 3D transformations.

Reviewer comment (Experiments):
It's also not obvious that block concatenation of multiple group representations, each with a certain multiplicity, is the best way to achieve it, and I believe this work could do with a bit more discussion on these points, and potentially further ablations/baselines of other ways one could implement the same intuition.

Author response:
Thank you for your suggestions. To address your concern, we additionally conducted experiments with Kronecker product-based GTA we describe below.  There are two conventional ways to construct representation matrices in representation theory: 1. the direct sum and 2. the Kronecker product of irreducible representations. The former is one we described in sections 3.1 and 3.2. 

The Kronecker product of two square matrices $A, B \in \mathbb{R}^{m\times m}, \mathbb{R}^{n\times n}$ is defined as 

$
A\otimes B = \begin{bmatrix}
    a_{11} B & \cdots & a_{1m} B \\\  \vdots  & \ddots  &\vdots \\\  a_{m1}B & \cdots  & a_{mm} B
\end{bmatrix} \in \mathbb{R}^{mn\times mn}
$

The important property of the Kronecker product is that the Kronecker product of two representations is also a representation (homomorphism):  $(\rho_1 \otimes \rho_2)(gg') = (\rho_1 \otimes \rho_2)(g) (\rho_1 \otimes \rho_2)(g') $ where $(\rho_1\otimes \rho_2)(g) := \rho_1(g) \otimes \rho_2(g)$. 
We additionally experimented with this Kronecker version of GTA on CLEVR-TR and MSN-Hard, where we use the Kronecker product of the $SE(3)$ representation and $SO(2)$ representations for $\rho_g$ in GTA:

\begin{align}
    \rho_g = \rho_{\rm cam} (c) \otimes (\rho_{h} (\theta_h) \oplus \rho_{w} (\theta_w)), \text{where $g = (c, \theta_h, \theta_w)$}.
\end{align}

The multiplicity of $\rho_{\rm cam}$, $\rho_{h}$, $\rho_{g}$ are set to 1, and the number of frequencies for both $\rho_{h}$ and $\rho_{w}$ is set to 4 on CLEVR-TR and 6 on MSN-Hard.
Below is the comparison between the Kronecker version of GTA (GTA-Kronecker) with the GTA described. Because both GTA-Kronecker and the original GTA encode the geometric structure in a homomorphic way, the performance is relatively close to each other, especially on MSN-Hard.  We added these results to Table 6 in the Appendix. 
| Method           	|  CLEVR-TR PSNR  | MSN-Hard PSNR |
|----------------------|:---------------:|:-------------:|
| GTA-kronekcer (ours) |   	38.32    	| 	24.52 	|
| GTA (ours)       	|	**38.99**	|   **24.58**   |

Reviewer comment (Experiments):
As suggested above, I would like to see a variant of GTA that does not involve block concatenation of the geometry representations matrices along with their multiplicites (to match the transformation to the input dimensionality `d`). Have the authors explored a variant such as: concatenate the flattened representation of all groups, followed by a linear projection up to `d`, element-wise multiplication with input features (with the rest of the GTA method as proposed)?

Author response:
Thank you for your question and suggestion of an interesting variant of GTA. We had not tested element-wise multiplication of linearly transformed flattened representations and additionally ran an experiment with that approach. Note that when we transform the representations into vector form multiplied by QKV vectors, we lose the homomorphism property: suppose $f(g) :=W vec(\rho(g))$ is the vector embedding of the representation $\rho(g)$ where $vec()$ is the flattening operation, then $f(gg’) \neq f(g)f(g’)$ in general. Thus we could say f is not faithful in terms of structure preservation of $g$. The table below shows that the performance significantly degrades with element-wise multiplication to QKV features (and outputs of QKV attn.). 

| Method|  CLEVR-TR PSNR  |
|-----------------------------|:---------------:|
| Element-wise multiplication |  	34.33   	|
| GTA (ours) |	**38.99**	|

We included this experimental result in Table. 5.

Reviewer comment (Theory):
The definition of $\rho_{g_i g_j}$ in Eq. 4 I believe is missing, which makes it difficult to understand how the authors go from Eq. 4 to Eq. 5, as well as their claims regarding the quadratic complexity in Eq. 4.

Author response:
Thank you for pointing this out. $\rho_{g_i g_j^{-1}} = \rho(g_i g_j^{-1}) \in \mathbb{R}^{d\times d}$ is a representation of the relative group element $g_i g_j^{-1}$ between $i$ and $j$-th tokens. From the homomorphism property of ρ, this can be decomposed into the product of the representations of each group element:  $\rho(g_i g_j^{-1}) =  \rho(g_i) \rho(g_j^{-1})$, with which we can derive the equation (5). (An example of the homomorphism: suppose $g$ is a degree of 2D rotation and $\rho(g): R(g)$ is a 2D rotation matrix given a degree $g$. Then for example, we have $R(40) = R(20+20) = R(20)R(20))$.
In the revised manuscript, we explicitly mention the homomorphism property after Eq. 4 and define the homomorphism property when we introduce group and representation in Section 3.

Reviewer comment (Experiments):
While some results are provided on real-world datasets, these are fairly limited in both size and quality (from looking at the videos). It's therefore difficult to conclude that this method is SoTA more generally in real-world cases. For instance, results would be strengthened if a study of the sensitivity of the model's predictions to camera errors during training and/or test time was included. This is relevant given that other methods in the literature have carried out these types of experiments (see e.g. SRT paper, Fig. 6).

Author response:
Thank you for your suggestion. We additionally train models with camera noise added to input views, which is the same setting as in the SRT paper. All modes are trained with σ = 0.1 and tested with different noise strengths on the CLEVR-TR dataset. We describe the experimental settings in detail in Appendix B.4.
The results are shown below. GTA shows better performance than RePAST regardless of the noise level.  The table below was added as Table 9. in the revised manuscript.
| σ                      			|  0.01   |	0.1 	|
|----------------------------- |:-------:|:----------:|
| (RePAST, wo/noise during training) |  33.80  |  30.57  |
| RePAST (Safin et al. 2023)  	|  35.26  | 35.14 |
| GTA (ours) |  **36.66**  |  **36.57**  |

Because of the limited time of the discussion period, we could not train on MSN-Hard for this experiment,
but will add results on that dataset in the final version.
Note that, RealEstate10k and ACID datasets include camera noise because each camera parameter is estimated with a structure from motion algorithm. We see even in those settings, the model outperforms Du et al. 2023, which uses APE-based positional encoding.