Paper: HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments (ICLR-2024)

Reviewer comment (Experiments):
I am curious about how hard are the three environments.

Author response:
> How hard are the three environments

The HAZARD challenge's difficulty primarily comes from its constantly changing environments. We believe it is challenging enough to effectively differentiate the performance of various methods. However, it is not excessively difficult, as all agents can complete the task to some degree. Specifically, the challenge involves planning in dynamic environments, comprehending the effects and dynamics of these changes, and accurately perceiving the evolving surroundings.

Reviewer comment (Experiments):
I do acknowledge that the authors try to provide versions of the benchmark that require visual perception of the environment. It would be nice though if a similar array of different options was provided on the action representation side. The current benchmark only supports very high-level actions. It is further unclear whether the RL policy that the authors compare to has all the same privileged observation and action primitive access that the LLM planner has. A fair comparison would be good here.

Author response:
Following your suggestion, we have provided interfaces for using low-level action APIs for agents, along with relevant documentation. However, when limited to low-level actions alone, current agents struggle with task completion. For example, when we replace the action set of wind scenes with low level actions {turn, move forward, pick up, drop} instead of {walk to nearest, pickup nearest, drop}, the RL agent almost fails to retrieve even one single object.

We preliminarily test the RL agent equipped with low-level actions in wind scenes. However, as the following results demonstrate, the agent almost fails in this setting, indicating that this new setting poses significant challenges to the agents.

| Without Perception     | Value | Step   |
|------------------------|-------|--------|
| RL                     | 8.5   | 1044.9 |
| RL (low-level actions) | 3.7   | 1144.0 |

Yes, the observations are the same. We modify the observation to fit a 2D-array for RL and text description for other methods, but they all have access to full observation. We modified the actions of RL a bit as described in the paper, as it’s infeasible for RL to output exact object indices.

Reviewer comment (Experiments):
While the current split of train:test is reasonable, it would be nice to investigate certain axis of generalization for the different agents more systematically. E.g. one could specifically test generalization to larger rooms or to new objects etc.

Author response:
In response to your valuable suggestion, we have created two new test sets: one with new objects and another with larger rooms. We've also provided documentation to facilitate the addition of new objects.

In addition, we propose a method to tackle the test set with new objects in the ‘with perception' version of HAZARD. We have replaced the R-CNN perception module with Grounded-SAM to generate segmentations for scenes containing new objects. We examine this new perception module on the test set with new objects, and results are summarized as follows:

| With Perception | Fire  |       |        | Flood |       |        |
|-----------------|-------|-------|--------|-------|-------|--------|
|                 | Value | Step  | Damage | Value | Step  | Damage |
| Rule            | 35.8  | 334.3 | 28.0   | 21.9  | 332.3 | 84.9   |
| Greedy          | 31.3  | 326.5 | 32.3   | 23.6  | 318.8 | 80.0   |
| Random          | 42.3  | 311.9 | 24.5   | 32.5  | 291.6 | 76.9   |
| RL              | 45.7  | 291.9 | 25.5   | 36.0  | 233.0 | 77.2   |
| MCTS            | 69.6  | 164.5 | 16.5   | 32.2  | 181.1 | 71.2   |

According to the results, with the help of the strong segmentation model, all baseline models are able to retain most of their effectiveness when encountering new objects. The results are included in Appendix C.

Reviewer comment (Experiments):
Finally, as acknowledged by the authors, the agent currently has no way of *influencing* the dynamics of the environment, e.g. by putting out the fires instead of working around them. It would be good to add such capabilities to more holistically evaluate disaster responses, but I acknowledge that this is challenging and the lack thereof does not invalidate the contributions of this submission.

Author response:
Thank you for your insightful comments. We are developing more diverse actions to enhance agents' interaction with the environment. For instance, we've introduced a 'put-out-fire' action, enabling an agent to extinguish fire at a specific location (please refer to the ‘put out fire’ gif on our website). We are also developing actions for water pumping and wind blocking. Such additions will not only increase mission complexity but also make the challenge more engaging.

Reviewer comment (Experiments):
The rendering quality and motion effects of the simulated scene are still somewhat different from reality, which may result in some perceptually based agents using camera images that will lead to domain gaps.

Author response:
Thank you for your comments. 

We view rendering quality as a balance with simulation speed. Given the large scale of fire and flood in the HAZARD challenge, we abandon pursuit for fine-grained simulations, such as detailed fire particle effects and fluid simulation, to evaluate embodied agents more efficiently. Additionally, our work utilizes the ThreeDWorld simulator, which comparatively offers realistic visual effects among current simulators.

Reviewer comment (Experiments):
The current scenario exhibits a limited scale. Is there a method available to efficiently expand the current scenario, such as loading assets and assigning attributes to different disaster scenes?

Author response:
Yes. Our codebase includes documentation on efficiently adding new assets and assigning attributes within HAZARD. We have updated the documentation with more detailed instructions for loading assets and assigning attributes.

Reviewer comment (Experiments):
The proposed LLM pipeline has to input the same task description at each step to perform sequential decisions, which is not efficient.

Author response:
We apologize for any confusion. In the chat model, the chain-of-thought reasoning's second step only requires previous conversations and an added prompt stating, “Answer with only one best next action. So the answer is option". For decisions at different time steps, it is necessary to input the task description each time, as LLM-based agents struggle to make decisions without the description in their context.

Reviewer comment (Experiments):
The mission objectives of the benchmarks are relatively simple, focusing on the task of object rescue, and the benchmarks can be expanded further in terms of mission complexity.

Author response:
Thank you for your insightful comments. We are developing more diverse actions to enhance agents' interaction with the environment. For instance, we've introduced a 'put-out-fire' action, enabling an agent to extinguish fire at a specific location (please refer to the current ‘put out fire’ gif on our website). We are also developing actions for water pumping and wind blocking. Such additions will not only increase mission complexity but also make the challenge more engaging.

Moreover, we wish to highlight that HAZARD's main difficulty lies in its dynamic environmental changes, which is our primary focus. Considering the baseline methods' performance on HAZARD, we believe the challenge already presents sufficient difficulty.

Reviewer comment (Experiments):
It seems that the hazard’s effect on the functionality of the embodied agents is not simulated. For instance, how does the fire and high temperature affect the functionality of the agent? Similarly, in the flood scenario, is the agent affected by the buoyancy force and drag force? The reviewer found the simulation of damage to the agents is an important piece of a realistic simulator.

Author response:
We value your suggestion and have accordingly integrated the impact of hazards on agents into the HAZARD challenge. To achieve this, we've developed a new setting where agents are directly affected by environmental effects:

* In fire scenes, the environment now causes damage to agents from fire by influencing the agent's temperature. This adaptation requires modifications to all baseline methods, particularly for fire scenes, as fire often blocks paths, demanding a different approach of planning algorithms.
* For flood and wind scenes, we've introduced a setting where agents are influenced by drag forces. Detailed descriptions and results of baseline methods in this new setting can be found in Appendix B. Given that most baseline methods show only a slight performance decrease, it suggests that environmental factors slightly increase the challenge in both flood and wind scenes.

Reviewer comment (Experiments):
How would the proposed approach address a partially broken embodied agent? Would approaches similar to Zeng [1] work?

Author response:
Currently, our approach to hazards can involve blocking paths or applying additional forces on agents, which do not result in partially broken embodied agents. Zeng [1]'s work examines different agent action impacts, distinct from our study of environmental change impacts on agent decisions. We have incorporated this discussion into our related work section.

Reviewer comment (Evaluation):
Could you elaborate why the MCTS-based method outperforms other baselines in terms of fire step and flood step?

Author response:
The MCTS method outperforms other baselines in both Step and Value metrics. Compared to LLM-based agents, MCTS-based methods consistently pursue the target with the lowest cost without altering their target during navigation or replan chances. On the other hand, LLM-based agents might choose a different target upon reaching the maximum navigation steps and having the opportunity to replan. Consequently, MCTS agents typically achieve a lower Step count than LLM-based agents.