Paper: Can Transformers Capture Spatial Relations between Objects? (ICLR-2024)

Reviewer comment (Evaluation):
It is unclear if the newer annotation scheme results in more accurate annotation or simply annotation that is better correlated with a relative location prior. Comparing Tables 4 and 11, one finds that all methods achieve a higher absolute score on the newer dataset, while the bbox baselines and the proposed method see stronger gains. Furthermore, Table 7 shows that the new annotation seems more specific regarding annotations being done in the labeler’s point of view or gravity to minimize ambiguity. As a result, while it is possible that the new definitions are more precise and a better evaluation, it is also possible that they are more strongly correlated with the relative locations in the image, which would introduce a strong prior. Some of the definitions in Table 7 support this point — for example, "A in front of B" is defined as A being closer to the camera than B. While this is a less ambiguous definition, it is a much easier one than the typical usage of "in front of," which often requires reasoning about the subject and object's relative orientation. For example, if two people are facing away from the camera, the further away person would be in front of the person closer to the camera. This is still unambiguous, but the prediction would be more difficult as it requires reasoning about relative depth as well as orientation. Said differently, the newer annotation scheme reduces ambiguity but also makes the relationships less dependent on context for cases like front/behind where RelatiViT shows the greatest performance improvements on the baseline (Table 6). This can be seen in the great improvements in performance of the bbox-only baseline. It would be useful to provide some qualitative analysis or quantitative comparisons that address this.

Author response:
**Q1:** It is unclear if the newer annotation provides a better evaluation or a simpler task. It is possible that the new annotations are more strongly correlated with the relative locations in the image, which would introduce a strong prior.

**Answer:** More precise definition does indeed result in an easier task. Imprecise definitions and the resulting inconsistent annotations lead to "irreducible errors" and therefore worse performance metrics. More importantly, the resulting models are less useful, because it is not clear what meaning to assign to the models if the task is specified in this imprecise/inconsistent way.

**Q2:** The newer annotation scheme reduces the ambiguity but it also makes the relationships less dependent on context. For example, the definition of “in front of” in the paper is a much easier one than the typical usage which often also requires reasoning about the subject and object's relative orientation.

**Answer:** Thanks for this example. In our opinion, this is only true for objects with "a canonical front". Not all object categories have such properties -- most don't. Furthermore, note that this involves reasoning through semantics. We do believe there is potential value for such a task, but it is worthwhile to decompose the geometric and semantic components of the task. i.e., first use our geometric notion of spatial relationships, then identify the semantic spatial orientations of the objects, and then combine these inferences into a "semantic spatial relationship" such as "human A is in front of human B".

Reviewer comment (Experiments):
The ablations are not very clear, and the conclusions drawn do not seem supported by the results. It is unclear how the method works without Pair Interaction, Context Aggregation, or Feature Extraction. For example, Pair Interaction is stated as the MLP that combines the two features — how is the prediction made without it? Additionally, how are predictions made without feature extraction — is it based on pixel values, patch embeddings, or something else? It is important to explain exactly what each ablated model looks like. Furthermore, the ablations are additive, so that the feature extraction ablation is applied to an already ablated model. While this is a valid experimental choice, it makes it difficult to understand the most critical modules as they are not ablated independently of each other. This is especially important in this problem where we know that feature extraction is not actually very important since bbox baselines perform very well, as shown in the paper.

Author response:
**Q3:** It is unclear how the method works without Pair Interaction, Context Aggregation, or Feature Extraction.

**Answer:** In our model, Pair Interaction is implemented by the attention mechanism between subject and object query tokens within the transformer. We ablate it by applying an attention mask on the attention weight between these two tokens. For Context Aggregation, similarly, we employ an attention mask on the attention weights between the query tokens and the image patch tokens. For Feature Extraction, we remove the ViT Encoder and relied solely on the patch embeddings of the image patches to predict spatial relations. We will describe the ablation experiment setup more clearly in our revision.

Reviewer comment (Experiments):
Several statements or claims are not fully supported or substantiated. The statement “it is difficult for CNN backbones to extract relation-grounded representations” (Sec 5.4) is strongly confounded by the use of positional embeddings. Based on the description, it seems that all methods (ViTs) have access to positional embeddings, while the CNN-Transformer baseline does not. While this is a common choice for CNNs, it appears to be a strong confounder here due to the strong performance of the bbox-only baseline. It is possible that ViTs outperform CNNs due to the explicit inclusion of positional embeddings rather than due to attention explicitly modeling pairwise relationships as suggested in Sec 4. Providing the CNN model with some positional embeddings, e.g., through CoordConv (Liu et al., NeurIPS 2018), could clarify this point and determine whether CNNs are still poor feature extractors even if another model performs context aggregation afterward (following design axes in Sec 4.1).

Author response:
**Q4:** Provide the CNN model with some positional embeddings, e.g., through CoordConv.

**Answer:** Thanks for your valuable suggestion! Following your advice, we incorporate positional embeddings into the input tensor of the first layer of ResNet in both DRNet and RUNet. The results, as illustrated in the table below, indicating that the CNN-based models (DRNet and RUNet) did not benefit significantly from the addition of positional embeddings. Their performance remains inferior compared to RelatiViT. This outcome reinforces our hypothesis that RelatiViT’s superior performance is largely due to its enhanced ability to capture spatial relationships through attention mechanisms, as opposed to the sliding convolution used in traditional CNNs.

|  Method   |  %Avg. Acc.  | %F1  |
| :----: | :----: | :----: |
| DRNet-CoordConv  | $76.86\pm0.96$ | $75.44\pm0.90$ |
| RUNet-CoordConv  | $75.91\pm0.73$ | $74.68\pm1.26$ |
| RelatiViT (ours) | $80.02\pm0.73$ | $78.32\pm1.01$ |

Reviewer comment (Evaluation):
The paper states that “[prior methods] are prone to simply predict the relationships based on the bounding box coordinates rather than learn the intended solutions by referring to the RGB input” (Sec 5.4), but that statement is not supported by the results. Most methods are consistently outperformed by the bbox-only baseline, suggesting that they are probably doing something other than predicting relationships solely from bounding box coordinates. While these models are likely learning shortcuts, it seems odd that this shortcut would be the use of bounding box coordinates if they are underperforming the bounding box baseline. Without further analysis, it is difficult to make such statements about what those models are doing.

Author response:
**Q5:** More analysis is required to make the statement that the prior methods tend to make predictions mainly according to bounding box coordinates.

**Answer:** Thank you very much for this suggestion. To verify whether the models predominantly rely on bounding box coordinates over image content, we conducted an intervention experiment by replacing the input images with random ones during testing. As shown in the following table, we report the *prediction flipping ratio*, which measures the frequency of predictions changing from True to False, or vice versa, across all samples. A higher flipping ratio suggests greater reliance on image content, while a lower ratio indicates a tendency to base predictions on bounding box coordinates. We find that DRNet and RUNet exhibit significantly lower flipping ratios than RelatiViT, indicating that RelatiViT places more emphasis on visual cues to when predicting spatial relations, contrary to the baselines focusing mainly on bounding box coordinates.

| Method | %Prediction flipping ratio |
| :----: | :----: |
| DRNet | $2.33 \pm 2.11$ |
| RUNET | $6.99 \pm 2.26$ |
| RelatiViT | $29.47 \pm 1.88$ |

Reviewer comment (Theory):
There may be a typo in the equation in Sec 3.1. It appears that the goal should be to minimize the negative log likelihood, but the current term minimizes the log likelihood — as for \( y = 1 \) the term would be minimized by estimating \( \hat{y} \) to be 0. If this interpretation is wrong, clarification is needed as to why this is the desired objective.

Author response:
**Q6:** Typo in loss function.

**Answer:** Thank you very much for pointing this out. We will correct it in our revision.

Reviewer comment (Novelty):
The spatial relationships examined in this paper are rather straightforward, as they do not encompass interactions between humans and objects or comparative relationships among multiple objects. This limitation might confine the applicability of the proposed method.

Author response:
Human-object interaction and multi-object relations are indeed promising topics to investigate. However, we find the existing methods struggle to accurately resolve even the fundamental spatial relations addressed in our paper. In this paper, we aim to investigate this spatial relation prediction task to show the importance and challenges of this under-explored field. Our intention is to provide a foundational framework that can serve as a springboard for tackling more complex scenarios in future research. 

As for the applicability, our method can be easily applied to the robot systems to query for spatial relations. This capability can be further leveraged for advanced applications such as PDDL-based planning or unsupervised reward learning, which will boost the performance of robot learning.

Reviewer comment (Evaluation):
As illustrated in Table 6, "RelatiViT" does not consistently outperform the "bbox-only" baseline across various categories of spatial relations. The potential advantages of "RelatiViT" might not be evident when taking into account the higher computational demands it imposes for image feature processing.

Author response:
In Table 6, our results are either comparable or outperform the bbox-only baselines across most categories. Moreover, in Table 10, RelatiViT performs better than bbox-only on 29 out of 30 predicate categories, which is a significant improvement. 

It is essential to recognize that while bbox-only models may achieve comparable accuracy in certain scenarios, such blind models, which do not leverage image inputs, possess inherent limits in further improvements. In contrast, our method leverages both visual data and advanced attention mechanisms, marking a notable advancement in predicting spatial relations. For the first time, our work demonstrates that vision-based methods can substantially exceed the bbox-only baseline. This achievement highlights the pivotal role of visual cues and relation modeling in this task. We believe that future research building upon our methodology will achieve even greater accuracy, far surpassing simpler baselines.

Reviewer comment (Evaluation):
The paper fails to elucidate the reasons why "RelatiViT" exhibits superior performance in the "left" category as opposed to the "bbox-only" baseline, while displaying subpar performance in the "right" category.

Author response:
There might be some misunderstanding. According to Table 6, the performance of RelatiViT is comparable with bbox-only on “left” and “right”. However, it's important to highlight that RelatiViT demonstrates significantly superior performance on more complex spatial relations such as "behind", "in", "under", and "front". These particular relations inherently require a more nuanced understanding of visual information.

Reviewer comment (Experiments):
Although it is good to unveil that the current methods and datasets have limitations on this new task, it is still imperative to do a thorough study on each proposed component to show why such a design is essential to the performance improvements. It is hard to be convinced about the significance of the contribution based on the current form.

Author response:
Thanks for your kind suggestion and for recognizing the good performance of our method. As also mentioned by Reviewer XTRZ and Reviewer ubwR, we have provided extensive experiments to analyze the success of RelatiViT in Section 5.5, Appendix A5, A7 and A8. In Section 5.5, we provide an in-depth ablation study, revealing that both context aggregation and good feature extraction are key contributors to the performance. The attention maps in Section 5.5 and Appendix A8 demonstrate that RelatiViT attends to appropriate regions to aggregate the context information for accurate spatial relation prediction. And the comparisons on each relation category in Section 5.5 and Appendix A5 show that RelaTiViT significantly surpasses baselines (especially bbox-only) in categories where visual information is critical. This indicates that RelatiViT successfully extracts the visual representations containing rich spatial relation information. Additionally, the prediction visualizations in Appendix A7 showcase that RelatiViT predicts the relations based on the visual information, e.g., object contact, occlusion, depth and background context, etc. This capability fundamentally contributes to its superiority over bbox-only baselines.

Reviewer comment (Experiments):
Besides linguistic biases during the annotation process, the statistics of spatial relations are biased, which could lead spatial relation detection to simply object classification. Rel3D addressed it by employing minimally contrastive construction, I would like to see authors' discussion on SpatialSense+.

Author response:
**Q2:** Discussion about the statistics bias of spatial relations.

**Answer:** Thanks for this insightful question. The original SpatialSense dataset is collected with adversarial crowdsourcing strategy. Specifically, the authors trained a simple model to predict spatial relations mainly based on the statistics, and then they asked crowdsourcing annotators to only label the samples that cannot be handled by this model. In this process, the statistics bias has been much alleviated. Our SpatialSense+ retains these samples and predicate categories from the original dataset. Consequently, we inherit this advantage of mitigating statistical bias.

Reviewer comment (Experiments):
Despite being titled as "Can Transformers Capture Spatial Relations between Objects", all prior baselines are CNN-based methods, it would be great to see authors include more advanced transformer-based models [1].

Author response:
**Q3:** Compare with more advanced transformer-based models.

**Answer:** Thank you for your insightful suggestions. We apologize for missing references such as SGTR [1] and RelTR [2]. Firstly, we want to clarify that these works are in different settings from ours: SGTR and RelTR aim to generate scene graphs and include an end-to-end detection process, but in our setting the detection is not required. Recognizing the distinct settings of our work, we make much effort to adapt RelTR to align with our experimental setting. This involved maintaining the same ResNet backbone as RelTR, encoding bounding box and object category information into learnable entity tokens, and modifying the relation prediction head to mirror the K-way binary classification head used in our models.

Upon implementing these changes, we observed an average accuracy of 57.91% with RelTR, which is markedly lower than anticipated. We attribute this underperformance to two primary factors: 

(1) RelTR relies on the features from only the last layer of ResNet, which we believe results in a significant loss of spatial information. 

(2) RelTR leverages learnable tokens to establish correspondence between the queries and the feature in region of interests, and then aggregate the information. This is difficult to achieve by the supervision of a small training set. Differently, our model utilizing large-scale pretrained transformer features and design a mask image based query localization strategy, which is able to establish the correspondence with fewshot training samples.

This experiment further highlights the capability to capture spatial relationships of our model. We will definitely include these references in our revision.

[1] Li R, Zhang S, He X. Sgtr: End-to-end scene graph generation with transformer[C]//proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 19486-19496.

[2] Cong Y, Yang M Y, Rosenhahn B. Reltr: Relation transformer for scene graph generation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

Reviewer comment (Experiments):
The paper mainly compared RelatiViT with ConvNet and ViT based networks. However, some of current multimodal Large language models (e.g. LLaVA [1], or even GPT-4V) seem be to capable of handling some spatial relationships. But I think it's acceptable to ignore these literature as they are very recent works and are not specially designed for this task.

Author response:
**Q1:** Compare with large vision language models.

**Answer:** Thank you for noting that our work precedes the publicly available large vision-language foundation models. It is indeed valuable to evaluate these models on spatial relationship prediction extensively, and we have performed some early tests on LLaVA. We evaluate the SpatialSense+ dataset on LLaVA latest version, i.e., LLaVA-v1.5-13b, and it only gets 50.27% overall accuracy, indicating that LLaVA   cannot handle this physically grounded spatial relation task. We think this result is very interesting and provides some inspiration for us about future work, so thank you very much again for this insightful suggestion.

Regardless of the specific results, for the foreseeable future, computation and time costs make it impractical for these foundation models to be directly applicable in many scenarios of interest, such as robotics with size, weight, power, and real-time operation constraints. We believe that it is valuable both practically for these applications and scientifically to understand and maximize the computer vision capabilities of smaller models for a large variety of tasks including ours.

Reviewer comment (Experiments):
The annotation of SpatialSense+ can be very hard to scale, since it requires annotators to understand precise definition of spatial relationships.

Author response:
**Q2:** The annotation of SpatialSense+ can be very hard to scale.

**Answer:** Thanks for your insightful question. We do not expect the labeling process to scale up, but want to fix this issue by taking advantage of large-scale unsupervised data and few-shot labeled data (i.e., self-supervised pretraining and finetuning). We admit that annotating such complex tasks cannot easily scale up, because it would definitely be very difficult and cost a lot of labor and financial resources. However, with the recent progress of foundation models, finetuning from large pretrained models makes it possible to inherit rich prior knowledge to handle complex downstream tasks with few-shot training sets. This has been a common solution for difficult problems in deep learning nowadays. Motivated by this, in this paper, we aim to fine-tune a good relation prediction model from the pretrained models with a few samples. In the future, especially in the age of large foundation models, we expect the solution to be reading out the relation information from foundation models by few-shot finetuning or even zero-shot prompting. Therefore, we do not require a large-scale spatial relation dataset, but only need a good pretrained model and a well-designed finetuning architecture and strategy, which is actually what we investigate in this paper.