Paper: STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning (ICLR-2024)

Reviewer comment (Experiments):
I have not found details about which relations have been conducted experiments on. ImageNet-VidVRD defines 132 predicates. Some of them are "static" but not "dynamic" spatial relations. Also, the spatial relations are connected with verbs e.g., swim_behind, and fly_behind. It is not clear how to handle the different definitions of spatial relations during pre-training.

Author response:
Thanks for pointing this out. We will add more details to the paper. 
In the ImageNet-VidVRD, we combine all similar relations (eg, behind, swim_behind, crawl_behing, fly_behind). "Swim_behind", for example, is just a subset of behind. 
As to how we handle static vs dynamic relations -- we convert all relations into "dynamic" representations, i.e., the same 3D model handles both dynamic and static relations. Static images are copied n times to create a 3D representation. Our class-wise experiments (in Table 2) suggest that the model can disambiguate between true dynamic 3D representations, and static 3D representations.

Reviewer comment (Experiments):
There are some uncovered spatial prepositions among the 30 defined spatial relations. It seems that they have been collected but not evaluated. Since STUPD is a synthetic dataset that suffers from huge gaps with real-world images, it is important to verify its effectiveness on real-world tasks.

Author response:
Unfortunately, there are no real-world visual relation datasets, to our knowledge, that cover the variety of relations that we have collected. This is, thus, a reflection for other researchers to expand their work to cover these relations as well. For current datasets that cover the limited number of spatial relations, it makes sense to only pretrain on the subset of STUPD that overlaps with the datasets. This helps us perform systematic ablations about the specific knowledge of individual relational meanings imparted by STUPD. 
As to whether or not the uncovered relations will be useful for real-life settings, our experiments provide strong arguments in that direction. 
We systematically showed that STUPD-pretrained models perform better on real-world datasets containing relations as we define in our paper. Since, the data designing and generation process is consistent across all prepositions, it is logical to deduce that overall, other prepositional relations in STUPD would also be useful to real-world settings.

Reviewer comment (Novelty):
I wonder about the necessity of proposing a new task of temporal relations. Their definition is detailed in Fig.2. Instead of a specific model to reason these temporal relations, it seems that we can directly apply temporal segmentation of each event and then compare the two segmented time stamps. For example, to judge ”A before B”, we first get the temporal segmentation A: $[t_1, t_2]$, B:$[t_3,t_4]$; if $t_2 \textless t_3$, then we say “A before B”.

Author response:
Actually your argument is correct. An oracle model with time-stamp based inputs is not really useful, since one can differentiate between many relations with simple rules. 
Based on another reviewers comment, we introduced another experiment (involving a VQA dataset), that shows how pretraining on temporal STUPD can also help in real-world settings. In this model, we evaluate whether models can learn temporal relations by analysing the videos, and understanding the nature of the individual events. We included many changes in the paper, all highlighted in the color blue.

Reviewer comment (Experiments):
I failed to find any experiment employing the STUTD dataset to improve the performance on real-world *temporal* relationship tasks.

Author response:
"Lack of use of the STUPD dataset as a temporal reasoning pretraining set.". You are right, and after several discussions, we decided to proceed with a pretraining experiment for Temporal-STUPD as well, which involves a VQA dataset. While most VQA datasets either focus on spatial relations only, or do not provide structured temporal relation annotations, we observe that the NeXT-QA dataset [1] is suitable for our needs. Another candidate -- the TEMPO dataset [2] would have been useful, but unfortunately the dataset has not been uploaded publically in its entirety by the authors. Please see Section 4.2 of the revised paper. Similar to Spatial-STUPD, we observe that Temporal-STUPD too helps in improving real world temporal relation reasoning significantly.

Reviewer comment (Experiments):
Lack of use of the STUPD dataset as a temporal reasoning pretraining set. Unlike what the authors mentioned at the end of Section 4.2, there is a lot of video-based work that focuses on temporal reasoning. For instance, many datasets [1,2,3,4] exist in the video question-answering domain. It would be better to employ some models that try to resolve the tasks suggested by such datasets.

Author response:
"Lack of use of the STUPD dataset as a temporal reasoning pretraining set.". You are right, and after several discussions, we decided to proceed with a pretraining experiment for Temporal-STUPD as well, which involves a VQA dataset. While most VQA datasets either focus on spatial relations only, or do not provide structured temporal relation annotations, we observe that the NeXT-QA dataset [1] is suitable for our needs. Another candidate -- the TEMPO dataset [2] would have been useful, but unfortunately the dataset has not been uploaded publically in its entirety by the authors. Please see Section 4.2 of the revised paper. Similar to Spatial-STUPD, we observe that Temporal-STUPD too helps in improving real world temporal relation reasoning significantly.

Reviewer comment (Experiments):
Reason for splitting the temporal relationship into ten categories. Based on what the author said, "before" is a subset of "by," "while" is a subset of "during," and "since" is a subset of "at" (from 1st para of Sec 3.4.2). Also, I don't think "by" (which implies a deadline) is used to describe the timing of two events in general. At least from my end, it is natural to say, "Turn in the assignment by midnight" instead of "Cut off the corners of the bread by the time you apply jam on the bread," for example. The author also mentioned 'redundant representation' in Section 2.1; in this regard, I failed to find any reason for keeping potentially redundant classes. Isn't it more natural to compress the classes into 7 instead of 10? If the author firmly believes a 10-class setting is much more meaningful, then I think it would be better to have an experiment in Point 4 but present the result with 7-class pretraining and 10-class pretraining.

Author response:
"Reason for splitting the temporal relationship into ten categories [is unclear]". The main goal of STUPD is to not just provide synthetic visual representations of common prepositional relations, but to also help models disambiguate between the different "senses" of a word, or in other words, the different meanings of the same word in different context. For example, "along" in Spatial-STUPD has two meanings in different contexts. Similarly in Temporal-STUPD, although "before" is a subset of "by", they have different subtle meanings. For example, "before" can be used to depict a relation between two events (event A before event B), or an event and a time point (Event A before Time B), but "by" is explicitly used to depict a relation between an event and a time point (Event A by Time B). (You also mentioned "Also, I don't think "by" (which implies a deadline) is used to describe the timing of two events in general". This is true, and has already been factored in the paper. See Figure 2 in the paper).

Reviewer comment (Presentation):
Formatting issue — \citet and \citep are different. Please carefully check the ICLR 2024 author's guidelines; It is improperly used in over 80% of the manuscript.

Author response:
citation format. Thanks for pointing this out. We have rectified this now.

Reviewer comment (Presentation):
Formatting issue — The author frequently used "<" and ">" without any escape character. Thereby, those symbols are repeatedly presented as flipped '!' and '?' characters throughout the text.

Author response:
"The author frequently used "<" and ">" without any escape character." This is corrected too.

Reviewer comment (Writing):
Typo — 2nd line of 2nd para of Intro (...in space of time" **pre**. Examples of...): I cannot understand what **pre** is for.

Author response:
"2nd line of 2nd para of Intro. I cannot understand what pre is for. ". This was a wrong citation, and has now been corrected.

Reviewer comment (Writing):
Typo — 50,000 uses the middle comma, but 5000 doesn't (always?) throughout the text.

Author response:
formating of large numbers with Commas. Corrected.

Reviewer comment (Experiments):
The synthetic data may not fully capture the complexity of real-world scenarios.

Author response:
We ensured that the selection of objects, and the nature of interaction was as realistic as possible. Not only did we use textured real-world-like objects, but restricted object pairs to only those that are likely to be seen in the real world. Also the physics engine used ensures that the interaction is natural.

Reviewer comment (Evaluation):
How does STUPD handle ambiguous or context-dependent prepositions where the spatial or temporal relationship might not be clear-cut?

Author response:
Different context dependent prepositions have distinct labels and videos corresponding to them. For example, the word "against" has two meanings, and different corresponding videos. Similarly, there are two different classes corresponding to "into". Within certain constraints, we cover all prepositional relations in our dataset. Maybe some less-frequently words are not covered, but their meaning is covered by one or the other of the classes in this dataset. There are 40 classes in total, and the full list can be seen in Appendix.

Reviewer comment (Experiments):
What measures are in place to ensure that the synthetic data in STUPD is diverse and representative of real-world scenarios?

Author response:
We started out by selecting a diverse range of objects from various supercategories (like vehicles, large grounded objects, small objects, larger objects, containers). Overall, the diversity of objects as well as the absolute number of objects is comparable to SpatialSense (a real world dataset). These synthetic objects have real-world-like textures. We also filter out object pairs that are unlikely to be seen in the real world (such as building on top of a car). 
Backgrounds are also selected so as to bring realness. Finally, a physics engine ensures that the interactions are real-world-like.