Paper: Task Planning for Visual Room Rearrangement under Partial Observability (ICLR-2024)

Reviewer comment (Evaluation):
The graph-based state representation requires shortest-path computation for all fully connected edges, but this seems quite computationally heavy, especially when we have a large number of nodes, leading to a drastically increasing number of edges.

Author response:
**W2 :** Our 2D occupancy map, with average dimensions of 24x13 grids, where each grid represents 0.25m, yields an average time complexity of approximately $3\times10^7 \approx 0.031s$ even with N = 100 objects, for computing all pairs' shortest paths using Dijkstra's algorithm. This time complexity is expressed as $\mathcal{O}(N \times N \times (V+E\times logV))$, where vertices V = 24x13 and edges E = 4x(24x13-24-13).

Reviewer comment (Novelty):
Some new metrics are introduced to measure agents' efficiency but they look a bit similar to SPL in navigation literature, which basically penalizes an agent's success rate by the length of trajectories it took so far. Similarly to the introduced metrics, as agents take more steps for rearrangement or search, SPL penalizes the success rates more, accordingly.

Author response:
**W4 :** Although SPL, SNS and ENR may seem similar in design, but serve distinct purposes and cater to a different problem. SPL metric is generally used in ObjectNav tasks, primarily penalizing the success rate of method for taking a longer path length. Whereas our SNS metric penalizes the agent for taking additional pick-place actions or search attempts in successful rearrangement scenarios with the same number of objects. ENR highlights the efficacy of our method in discovering unseen objects to overcome partial observability. ENR is simply the ratio of Number of unseen objects initially to the number of steps required to search them.

Reviewer comment (Experiments):
The environments used in the proposed benchmark look quite "clean." It seems that we have objects only related to rearrangement tasks, as illustrated in Figure 16 and the supplementary video. This looks quite far from practical scenarios as we usually have many objects inside rooms.

Author response:
**W5 :** To ensure clarity in the video illustration, we intentionally created clean scenarios. Typically, the RoPOR benchmark dataset generates scenes with many objects, but only a few of them require rearrangement.

Reviewer comment (Presentation):
It might be better to divide a result table (e.g., Table 1) into one for the main results and the other for the ablation study for better readability.

Author response:
**W6 :** Table 1 presents a comparative study with prior works on RoPOR, while Table 2 provides a similar comparison on RoomR. Additionally, Table 3 showcases the ablation study conducted on RoPOR.

Reviewer comment (Writing):
Convoluted approach that is a bit hard to follow, especially because many details are omitted in the main paper. It would also be helpful to more clearly define the task setup (inputs and outputs and their explicit representations).

Author response:
**W1 :** Task setup is described in Sec 2 introduction & Sec 2.1. To alleviate the reviewer's concern, we will improve the presentation in the camera-ready version.

Reviewer comment (Theory):
Not obvious why the SRTN needs to be learned instead of manually defined.

Author response:
**W2 & Q4 :** The hand-labelled search's efficacy is limited to scenarios mentioned within it, whereas our Search Network module generalizes well to search unseen objects outside the training dataset due to the enormous semantic knowledge in LLM (shown by the seamless transferability to Habitat in supp video 19:17-20:26).

Reviewer comment (Experiments):
Manual heuristics (where objects are unlikely to be found) and GT information (e.g. odometry, collision vector, etc.) is required.

Author response:
**W3 :** No, manual heuristics are not required because the commonsense priors can be learnt from the train set of RoPOR. All prior works assume the GT information such as RGB-D and egomotion.

Reviewer comment (Experiments):
Critical ablations are not in the main paper.

Author response:
**W4 & W5:** We will reconsider the space management in the main paper and shift the reward structure and critical ablations for the camera-ready version.

Reviewer comment (Theory):
Assumes perfect motion and manipulation.

Author response:
**W9 :** This is a standard assumption in all the previous works.

Reviewer comment (Experiments):
Goal definition of having the goal scene already set up and utilizing an exploration stage is a bit impractical, as it requires setting up the goal scene every time it is changed. Also the robot needing to explore the goal state takes time compared to, say, language defined goals.

Author response:
**W10 :** To perform rearrangement, the robot needs to know the tidy goal state, which is captured using a one-time goal state exploration. This approach is inline with state-of-the art methods. Yes, in the future we will study the integration of language-defined goal state, as it requires us to re-purpose the Search network to predict the location of objects from the language based goal, keeping our planning module intact.

Reviewer comment (Evaluation):
What are the failure cases? Does the agent ever get stuck in a loop?

Author response:
**Q1 :** In certain scenarios, the perception used in our method results in erroneous object detection. This leads to unnecessary search efforts and ineffective sequencing due to inaccuracies in object positions, causing failures in object pick/place actions. Another failure case arises when an unseen object is occluded by cluttered static objects - those that will not be moved during rearrangement.

Reviewer comment (Experiments):
Are duplicate objects handled (e.g. multiple sponges in a scene)? Are these included in the test data, and how does the agent perform under these conditions?

Author response:
**Q2 :** Yes, duplicate cases are handled and are a part of the test data. In both the goal and current states, our object lists uniquely represent each object by incorporating their semantic label and 3D centroid position. Therefore, duplicate objects are treated the same as other objects and exhibit similar performance.

Reviewer comment (Experiments):
SRTN requires manually defined rules and heuristics about where objects are unlikely to be found (e.g. cup in bathtub). Is the network unable to learn these probabilities automatically?

Author response:
**Q3 :** To learn automatically, the agent needs to explore all the untidy scenes in the train set of RoPOR and create a knowledge graph, which would then be used to finetune the LLM embeddings. This process is tedious and would not effectively capture all the plausible relationships, leading to poor generalizability. Therefore, we train SRTN to finetune the LLM embeddings using the manually defined data in the Search Network Dataset (SND), consisting of approximately 24,000 annotations where each object and room-receptacle pair is ranked by 10 individuals.

Reviewer comment (Experiments):
Does the SRTN need to be learned via the MLP? It seems like the probabilities can be pulled directly from the data without learning. We could simply build a hardcoded table of probabilities. For example, the probability of finding a sponge in the sink can be hardcoded to 0.80. Is there a reason why that number would change? What is the benefit of learning here?

Author response:
**W2 & Q4 :** The hand-labelled search's efficacy is limited to scenarios mentioned within it, whereas our Search Network module generalizes well to search unseen objects outside the training dataset due to the enormous semantic knowledge in LLM (shown by the seamless transferability to Habitat in supp video 19:17-20:26).

Reviewer comment (Reproducibility):
How does the agent go from the output of the planner to discrete motions (rotate, move forward, pick up, place)?

Author response:
**Q5 :** Similar to all the previous works, we assume the availability of default navigation and motion planning. To generate a path from point A to B, we use the Djikstra's algorithm on the 2D occupancy map. Subsequent execution of low level action (such as Move forward or rotate right or pick-place) is performed using the built-in functions of Ai2Thor.

Reviewer comment (Theory):
How is navigation performed when going from point A to point B? Is it assumed that this problem is solved?

Author response:
**Q5 :** Similar to all the previous works, we assume the availability of default navigation and motion planning. To generate a path from point A to B, we use the Djikstra's algorithm on the 2D occupancy map. Subsequent execution of low level action (such as Move forward or rotate right or pick-place) is performed using the built-in functions of Ai2Thor.

Reviewer comment (Experiments):
There is already an existing dataset (and associated metrics) for the same setup in Sarch et al. (Sec. 4.5 of their paper). While there are some deviations between the proposed dataset and the existing dataset, it still makes sense to benchmark the proposed approach against the already existing dataset that has baselines already benchmarked against it. Is there a reason this wasn't done? The setup of the existing seems like a subset of the proposed dataset's setup, so it seems doable.

Author response:
**W6 & Q6 :** Yes, we have evaluated our method in existing dataset RoomR. Please find the comparative study in Table 2 and Sec B.

Reviewer comment (Evaluation):
**[Major]** How can the Search Network leverage prior knowledge as the room is untied? More specifically, if object shuffling/placement is done randomly, shouldn’t there be no prior remaining? Indeed, in phase 2, objects are placed in locations were there shouldn’t be (so that the agent can re-organize them). It is hard for me to understand how, in this case, the Search Network can learn anything meaningful. A result in the ablation study (Table 2) seems to confirm this intuition: the performance of Ours-RS, where the Search Network is replaced with a uniform random sampling of the next receptacle to visit, is very close (1p Success Rate) to the performance of Ours. The difference does not seem to be significant enough to claim the Search Network does more than random search. In order to claim a gain, authors should report the mean and standard deviation over a few training runs (random seeds).

Author response:
**W1 :** Refer to the response in the common concern section. 
Using SNS to gauge the search efficacy of Ours and Ours-RS is not entirely correct. Although, Ours-RS searches the receptacles uniformly, but it can eventually find the object through exhaustive search. To demonstrate this limitation, we have introduced two other metrics in the paper - ENR and ATC. ENR represents the ratio of initially unseen objects to the total search attempts needed to locate them. In Table 3 of the Appendix, the ENR values for Ours and Ours-RS demonstrate a widening gap, approximately 18% for 5 objects and increasing to about 22% for 20 objects. Moreover, the ATC difference between Ours and Ours-RS ranges approximately 30% to 50%. These metrics underscore the significance of search efficiency in Ours compared to Ours-RS.
- **W1.1 :** Sec J and Table 3 in Appendix shows the performance of Ours - RS on the RoPOR dataset.
- **W1.3 :** Refer the response in the common concern. For a fair comparison with state of the art methods, we have duly trained them in the train set of our benchmark RoPOR before evaluation. The benchmark dataset RoPOR is split into train, validation and test in the ratio of 70:10:20.

Reviewer comment (Experiments):
**[Major]** The authors should provide more details about how objects are moved in phase 2: there should not be any prior remaining, and if there is, it might mean that the comparison is unfair with other methods because the authors’ search model might have been trained to learn those “shuffling priors” while it is probably not the case of previous work.

Author response:
**W4 :** The sorting network just classifies room-receptacles for objects into three groups based on the likelihood of finding the object at each location. However, it does not assign scores to rank them, consequently failing to prioritize the search in the most probable receptacle first. Without the scoring network, the search strategy must randomly select a room-receptacle from the higher probability class. This random selection degrades efficiency in terms of ENR and ATC, increasing the number of search attempts, as indicated in the ablation table below.

**With SCN V/S Without SCN**
| Number of objects | Unseen |Objects||With SCN|||| Without SCN  ||
|:----: | :-:| :-: |    -:  |   :-:  |   :-  |-|   -:  | :-:  |   :-  | 
|     |  OOF  | OPR| SNS&#8593;| ENR&#8593;| ATC&#8595;|| SNS&#8593;| ENR&#8593;| ATC&#8595;|   
|     5      |     2    |      0        |     0.60             |      0.48          |       14.33         | |      0.52          |         0.41       |     19.57           |  
|             |     0     |      2          |      0.58            |      0.47         |       14.89         | |     0.49           |      0.38          |        20.83        |
|     10      |     4     |      0         |      0.64            |      0.53          |       25.56         | |   0.54              |      0.45          |      31.17          |  
|             |     0     |      4         |      0.62          |      0.52           |       25.97           | |       0.52         |           0.39     |        32.11        |

Reviewer comment (Experiments):
**[Major]** One might argue that the Sorting Network only could be enough to predict the most likely object-receptacle pairs. Authors should provide an ablation study showing the impact of the additional Scoring Network.

Author response:
**W4 :** The sorting network just classifies room-receptacles for objects into three groups based on the likelihood of finding the object at each location. However, it does not assign scores to rank them, consequently failing to prioritize the search in the most probable receptacle first. Without the scoring network, the search strategy must randomly select a room-receptacle from the higher probability class. This random selection degrades efficiency in terms of ENR and ATC, increasing the number of search attempts, as indicated in the ablation table below.

**With SCN V/S Without SCN**
| Number of objects | Unseen |Objects||With SCN|||| Without SCN  ||
|:----: | :-:| :-: |    -:  |   :-:  |   :-  |-|   -:  | :-:  |   :-  | 
|     |  OOF  | OPR| SNS&#8593;| ENR&#8593;| ATC&#8595;|| SNS&#8593;| ENR&#8593;| ATC&#8595;|   
|     5      |     2    |      0        |     0.60             |      0.48          |       14.33         | |      0.52          |         0.41       |     19.57           |  
|             |     0     |      2          |      0.58            |      0.47         |       14.89         | |     0.49           |      0.38          |        20.83        |
|     10      |     4     |      0         |      0.64            |      0.53          |       25.56         | |   0.54              |      0.45          |      31.17          |  
|             |     0     |      4         |      0.62          |      0.52           |       25.97           | |       0.52         |           0.39     |        32.11        |

Reviewer comment (Novelty):
**[Major]** What is the interest of this Proxy Reward network? The paper mentions it is a way to predict a dense reward to train the RL agent. However, given a simulator, couldn’t we simply compute a dense reward from privileged simulator information at training time?

Author response:
**W5.1 :** The motivation for proxy reward network is discussed comprehensively in the last paragraph of Sec 2.4.1 and Sec 2.4.2. Yes, we utilize a hierarchical dense reward structure, as stated in Sec D, to populate the replay buffer with step-wise reward for every episode. However, recent studies (Ren et al.) indicate that this handcrafted reward system performs well in scenarios within its intended design but struggles to generalize to extraneous cases. In addition, the dense reward structure provides feedback to the RL solely based on the outcome of each step, lacking consideration for the episodic objective. This leads to reduced sample-efficiency during RL training as shown in Fig 3 (paper). Therefore, we use proxy reward network to learn the reward distribution of the episodes from the experience data of the replay buffer, to give our agent a notion of the overall objective of the episode.

Reviewer comment (Reproducibility):
**[Major]** What is the training ground truth for the Proxy Reward network?

Author response:
**W5.2 :** The training ground truth for the proxy reward network is the scaled episodic reward of the randomly sampled episodes from the replay buffer. The episodic reward is the sum of rewards for all the steps in the episode. It is scaled with the ratio of Number of steps in the episode to the total path length cost for that episode, as mentioned in Eq 6. The training of the proxy reward occurs concurrently with the RL training, as explained in Sec 2.4.2 and Algorithm 1.

Reviewer comment (Evaluation):
**[Major]** What is the average return on the y-axis of Figure 3? Such return is indeed associated with a specific reward function: what are the terms of this reward function? Moreover, I would like the authors to provide more details about the comparison done in Figure 3 and thus the conclusions we can draw from it.

Author response:
**W5.3 :** Average return is the mean of rewards of the last 10 steps during RL training. We use a hierarchical dense reward structure, elucidated in Sec D of Appendix. In Figure 3, the number of training steps required by RL to consistently achieve a high average return is illustrated for different reward mechanisms, namely sparse, dense, proxy reward network trained with RRD, and CB-RD. It clearly establishes the sample-efficiency in training of our novel Cluster Biased - Return Decomposition (CB-RD) method, showcasing the highest return in the fewest training steps.

Reviewer comment (Experiments):
**[Major]** The paper mentions the introduced method “assumes the availability of perfect motion planning and manipulation capabilities”. While this is a strong assumption, it does not outweigh the contributions in this work. However, an important question is: Are all the baselines this method is compared against also benefiting from the same assumption? Otherwise, this could be considered as an unfair comparison.

Author response:
**W6 :** Yes, these assumptions are in line with all the baselines.

Reviewer comment (Novelty):
**[Minor]** When introducing their SNS metrics, authors should cite *Anderson et al., On Evaluation of Embodied Navigation Agents* that introduced quite similar metrics such as SPL.

Author response:
**W7 :** Sure, we will add this citation in the camera ready version.

Reviewer comment (Writing):
**[Minor]** Paper citations are not properly inserted in the text. Authors should use parentheses (\citep{} in Latex) when needed, and remove double citations (e.g. “Sarch et al. Sarch et al. (2022)”).

Author response:
**W8 :** We will rectify this in the camera-ready version.