Paper: MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The proposed MANGO constructs a simplified text-world, its connection to real-world navigation and mapping of embodied agents is unclear. Specifically, it assumes a known environment but many real-world navigation is only partially-observed.

Author response:
Our MANGO environments are partially-observed as well: at each step, an LLM only sees the current view and "remembers" what it has seen in the past (i.e., walkthrough) but has no access to the full map. Further, MANGO requires navigational reasoning due to the existence of imputed edges (see Sec 2.2 in paper).

Reviewer comment (Experiments):
The proposed MANGO constructs a simplified text-world, its connection to real-world navigation and mapping of embodied agents is unclear. Specifically, the structures of spaces and agents' actions in MANGO are very simple, whereas in the real-world they are often very diverse and complex.

Author response:
Yes, our benchmark is more simplistic than real settings. Please see [General Response - Practicality and Impact] for our detailed discussion about this. 

We will also discuss this limitation in the new Limitations section.

Reviewer comment (Experiments):
The proposed MANGO constructs a simplified text-world, its connection to real-world navigation and mapping of embodied agents is unclear. Specifically, it only provides text data and it is hard to extend to visual inputs (consider the emerging large VLMs for addressing similar problems).

Author response:
Yes, our benchmark is more simplistic than real settings. Please see [General Response - Practicality and Impact] for our detailed discussion about this. 

We will also discuss this limitation in the new Limitations section.

Reviewer comment (Experiments):
Section 2.2 and Appendix — it is unclear why slightly different data is applied to evaluate different LLMs, and the paper does not provide a clear explanation.

Author response:
Because each LLM may read a walkthrough of a slightly different length due to their technical (e.g., tokenizer) differences. When the walkthrough gets shorter, some questions may become unanswerable since they involve locations and paths that are no way inferred from the given shorter walkthrough.

Reviewer comment (Evaluation):
Apart from the proposed metrics for DF and RF, the authors could consider whether some navigation-oriented measurement might be helpful, e.g., Success weighted by Path Length (SPL) (On Evaluation of Embodied Navigation Agents. Anderson et al., 2018).

Author response:
This is a sweet idea! Thank you. 
We will add it in the camera-ready version.

Reviewer comment (Experiments):
The paper does not provide any results on experimenting with different prompts for the LLMs, nor any insight on how to write those prompts.

Author response:
We only lightly tuned the prompts. 

In general, we follow the guidelines from CoCoGen [1] and CodeIE [2] to write the instructions, formulating the prompts in a structured format and asking the LLM to generate structured outputs. 

There are small wording variations across models (e.g., "nodes" better for some models, but "locations" better for others; cases; !!! for emphasizing; etc). 

We will publish all our prompts and experiment logs after the paper is published.

Reviewer comment (Evaluation):
It is unclear whether the results shown in Tables are from a single run of the LLMs or from multiple runs and averaged.

Author response:
Single run, but we set the temperature to be 0 for reproducibility.

Reviewer comment (Experiments):
It is unclear how tuning LLMs on MANGO, e.g., using an adaptor for low compute cost, would affect the results, and whether the results would become much better.

Author response:
Yes, we think it will be better. 

The primary goal of our MANGO benchmark is to function as a testbed for 0-shot ability of LLMs in mapping and navigation. In other words, it is test-only. We regard this as the most interesting thing to track since 0-shot transferability is a fundamental property of intelligence. In this regard, we are similar to other test-only benchmarks such as MMLU [3].  

But we agree that it is interesting to investigate how low-cost-adapting can improve the performance. In particular, an interesting investigation is how an LLM can learn from a few mazes and then transfer to the others. We will list this as a future work in our camera-ready.

Reviewer comment (Experiments):
The proposed dataset does not effectively test mapping and navigating abilities, as the samples can be easily converted into a graph with locations as nodes and directions as relations. This is too simplistic for most real-world robotics scenarios, which involve more complex object and position relationships. For example, the robots in a house, or robots (cars) on the street may be facing much more complex scenes.

Author response:
We agree that our benchmark is simpler than your examples, but it is still challenging for best-to-date LLMs and functions as a testbed where interesting analysis can be done. 

But please note that it can not be easily converted into a symbolic graph since language-to-structure mapping is not easy to learn. Please read [General Response - Have We Evaluated Strong Methods?] for more discussion.

Reviewer comment (Novelty):
The simplicity of the current dataset means it could be solved by an agent translating natural language into <source, path, destination> triples, then using code or search libraries. The natural language is generated by patterns, making natural language understanding easy. This paper may have limited research impact, as future studies might follow the path of the GSM8K dataset, using methods such as PAL or LLMs with code interpreters as tools.

Author response:
First, MANGO requires more than name entity *recognition*. According to Sec 2.1 and A.2 in paper, what MANGO requires is more of name entity *resolution*:  
- the location names are often embedded in very rich contexts; 
- the location names are often not *explicitly* mentioned in the descriptions; 
- multiple distinct locations may share names (e.g., "Hall") in descriptions so that our questions have to distinguish them with additional descriptors (e.g., "Hall (1st floor, north end)"). 

Under the above conditions, it will be difficult to extract compatible triplets: 
- each triplet may look reasonable individually and separately; 
- but consecutive triplets can not be seamlessly connected; 
- or distinct locations may be extracted as the same node in the graph. 

Second, LLMs will still make mistakes even when the given information is clean and clear. 

We are working on new experiments to resolve this concern of yours. In case the new experiments do not finish in time, here we share an anonymized chat history with GPT-4 to show how difficult it is to obtain a correct map from NL descriptions: https://chat.openai.com/share/6d2d3d9b-b992-410f-8ce7-0d55bfba01bf 

Note that it says "ChatGPT" but the purple logo means that its backend is GPT-4. 

Also note that: 
- it made mistakes; 
- we instructed it to make corrections, with clear guidance; 
- it still failed. 

We are systematically running a similar experiment on MANGO walkthrough, and we will update you with the accuracy. 

Does this resolve your remaining concerns?

Reviewer comment (Novelty):
Can this task be addressed using traditional search algorithms like depth-first-search or breadth-first-search?

Author response:
We did it for you! Please read [General Response - New Results of Symbolic Graph] for details. We hope that this set of results resolves your concerns. We will add them in the camera-ready. 

In addition, we wrote the [General Response - Why NOT Synthetic Data or Generated Graphs] to reemphasize the focus and value of the current version of the paper. 

Have we resolved all your concerns now?

Reviewer comment (Novelty):
While the experiments in the paper do demonstrate that the task is challenging for LLMs, it is unclear for me how the proposed benchmark differs from all other datasets in the literature.

Author response:
> how the proposed benchmark differs from all other datasets in the literature. 
> How does this compare to datasets used in embodied AI and NL navigation such as ALFRED? 
> Does the dataset reveal strength/weakness of the LLMs that is overlooked on other datasets?

Please see [General Response - Novelty]: basically, our MANGO focuses on an important dimension---i.e., mapping and navigation---which is different from the foci of other datasets.

Reviewer comment (Experiments):
Is it necessary to derive the dataset from real textgames? Would it work to use pure synthetic data, similar to how some of the symbolic reasoning datasets such as SCAN are created.

Author response:
> It is unclear whether the challenge for the model lies in abstracting the map / environment from the NL description, or in memorize / trace the map during reasoning. 

It would have been unclear if we only experimented with easy DF and RF questions, and only experimented with the original setting. 

Note that the "simplified" setting removed all the descriptions but only kept the location names, which is basically a symbolic walkthrough of map. Results in Table-1 show the performance differences between the "original" and "simplified" settings: in general, simplifying the walkthrough improves the accuracy, indicating that "abstracting map from NL" adds a layer of difficulty. 

Also note that the hard questions involve edges between locations that logically exist but are never mentioned in the walkthrough. It means that an LLM can *not* answer them via simply memorizing and tracing seen steps, thus having to perform deep navigational reasoning. 

If this discussion hasn't addressed your concern, could you please be specific about what kinds of synthesized data would do something more than what our MANGO benchmark can do? Or, more precisely, could you kindly give one example of a synthesized / symbolic map with a DF or an RF question that you think can test a deeper level of mapping and navigational reasoning that our MANGO benchmark can not do? We feel that MANGO has actually been doing what you expect a synthesized map can do, but we were just talking a little past each other. So we would appreciate a concrete example about which we can continue a deeper discussion. Thanks! 

In addition, per R.M8P4's request, we are already committed to publishing a version of our data where each location is associated with a unique ID and questions only mention IDs. That version is essentially a synthesized map except that the maps are carefully crafted by human experts. 

p.s., maybe not most important, but why do you think SCAN is a good comparison with MANGO? SCAN focuses on *execution* of a given sequence of moves, while our MANGO focuses on *finding* a sequence of moves; these are relevant but orthogonal tasks, aren't they?

Reviewer comment (Experiments):
How does this compare to datasets used in embodied AI and NL navigation such as ALFRED?

Author response:
> how the proposed benchmark differs from all other datasets in the literature. 
> How does this compare to datasets used in embodied AI and NL navigation such as ALFRED? 
> Does the dataset reveal strength/weakness of the LLMs that is overlooked on other datasets?

Please see [General Response - Novelty]: basically, our MANGO focuses on an important dimension---i.e., mapping and navigation---which is different from the foci of other datasets.

Reviewer comment (Evaluation):
Does the dataset reveal strengths/weaknesses of the LLMs that are overlooked on other datasets?

Author response:
> how the proposed benchmark differs from all other datasets in the literature. 
> How does this compare to datasets used in embodied AI and NL navigation such as ALFRED? 
> Does the dataset reveal strength/weakness of the LLMs that is overlooked on other datasets?

Please see [General Response - Novelty]: basically, our MANGO focuses on an important dimension---i.e., mapping and navigation---which is different from the foci of other datasets.

Reviewer comment (Experiments):
The authors conduct extensive experiments on GPT-3.5 and GPT-4 which is very helpful. However, instead of focusing on models that already perform well, I feel the paper could benefit from more experiments on why the other models perform much worse than GPT models, although they have demonstrated great performance on other tasks.

Author response:
> Why the other models perform much worse than GPT models, although they have demonstrated great performance on other tasks. Why for models like RWKV and LLAMA the size does not seems to affect the performance much. Are specific model designs, such as attention, position embedding, instruction tuning, affect the performance?

We would love to analyze these, and please see [General Response - New Results]. 

Our new results include the effects of different tuning methods.

Reviewer comment (Evaluation):
Why for models like RWKV and LLAMA the size does not seem to affect the performance much.

Author response:
> Why the other models perform much worse than GPT models, although they have demonstrated great performance on other tasks. Why for models like RWKV and LLAMA the size does not seems to affect the performance much. Are specific model designs, such as attention, position embedding, instruction tuning, affect the performance?

We would love to analyze these, and please see [General Response - New Results]. 

Our new results include the effects of different tuning methods.

Reviewer comment (Experiments):
Are specific model designs, such as attention, position embedding, instruction tuning, affecting the performance?

Author response:
> Why the other models perform much worse than GPT models, although they have demonstrated great performance on other tasks. Why for models like RWKV and LLAMA the size does not seems to affect the performance much. Are specific model designs, such as attention, position embedding, instruction tuning, affect the performance?

We would love to analyze these, and please see [General Response - New Results]. 

Our new results include the effects of different tuning methods.

Reviewer comment (Experiments):
In the paper the LLMs are prompted to directly generate the solution, with a small amount of COT reasoning. This is actually different from how humans solve the task, where oftentimes we need to parse the walkthrough and draw the map first. This is also how many embodied agent/NL navigation works have built up the system. I would expect a baseline in this direction to have much better performance.

Author response:
> need to parse the walkthrough and draw the map first… I would expect a baseline on this direction to have much better performance. 
> small amount of COT reasoning… parse the walkthrough and draw the map first… a baseline on this direction to have much better performance.

Please see [General Response - Have We Evaluated Strong Methods?].

Reviewer comment (Experiments):
Is there a comparison between the proposed benchmark with pure synthetically generated mazes to demonstrate the value of constructing the dataset from real textgames?

Author response:
> Is it necessary to derive the dataset from real textgames? Would it work to use pure synthetic data, similar to how some of the symbolic reasoning datasets such as SCAN are created.
> Is there comparison between the proposed benchmark with pure synthetic generated mazes to demonstrate the value of construct the dataset from real textgames?

Please see [General Response - Practicality and Impact].

Reviewer comment (Experiments):
I will like to see the benchmark performance for random responses (within a class bound) to verify the amount of information gained by base model.

Author response:
> benchmark performance for random responses (within a class bound) to verify the amount of information gained by the base model. 

We had random results in an earlier version but removed them because they were too bad. We can add them back to the appendices, but we could assure you that those numbers are nearly 0 compared to the results of the LLMs.

Reviewer comment (Experiments):
Need some explaination regarding para before 3.4 in terms of pilot experiments.

Author response:
> Need some explanation regarding para before 3.4 in terms of pilot experiments. 

That entire paragraph is explaining the pilot experiments, isn't it? 

If what you'd like are the detailed numbers, we can surely add them in the appendices of the camera-ready version. 

Otherwise, can you be specific about what precisely in this paragraph needs clarification so we can make more targeted revisions? Thank you very much!

Reviewer comment (Presentation):
Related work should be broken into sub-sections of research topics - the current norm - for ease of readibility.

Author response:
> Related work should be broken into sub-sections of research topics - the current norm - for ease of readability. 

Breaking into subsections with headings is a nice idea and we know many papers do it. 

In our Related Work, each paragraph is about a different field or topic; it just does not have a boldfaced heading. This way is also widely adopted by many well-written (and even award-winning) papers. 

We prefer our current style because it reads more smoothly (i.e., not being interrupted by headings). 

This taste of style is very subjective. Perhaps a different writing style should not be considered as a "weakness"?

Reviewer comment (Presentation):
The purpose of Fig. 6 which is too much info is not clear.

Author response:
> The purpose of Fig. 6 which is too much info is not clear. In page 7, what makes the maze challenging needs some results to support the text descriptions below. 

Fig-6 is similar to Fig-3 but its metric is reasoning accuracy. The design of Fig-3 is explained in Sec 3.3. Results supporting page-7 are in Tab-2, which is referred to in page-7 ("Table 2 displays the regression results.").

Reviewer comment (Evaluation):
In page 7, what makes the maze challenging needs some results to support the text descriptions below.

Author response:
> The purpose of Fig. 6 which is too much info is not clear. In page 7, what makes the maze challenging needs some results to support the text descriptions below. 

Fig-6 is similar to Fig-3 but its metric is reasoning accuracy. The design of Fig-3 is explained in Sec 3.3. Results supporting page-7 are in Tab-2, which is referred to in page-7 ("Table 2 displays the regression results.").

Reviewer comment (Novelty):
The use cases and applicability in real life scenarios like robotics is not well established, requesting to look to the plethora of work in embodied intelligence and adapting the problem in that regard.

Author response:
> The use cases and applicability in real-life scenarios like robotics is not well established, requesting to look to the plethora of work in embodied intelligence and adapting the problem in that regard.
> How fruitful are game environments to real life human occupied or industrial environments? Is the transfer easily equitable? 
> How will the system evolve if vision language based models like CLIP need to be tested - as that is more practical? 

There is a lot of interest in the robotics community in using language as a medium decomposing tasks into a sequence of subtasks that better match the primitives available to robots. We see mapping and navigation as one such example. Indeed, our results may not be immediately applicable to real-world embodied agents. However, as people in the robotics community increasingly rely on available LLMs for reasoning, there is merit in understanding the capabilities of the off-the-shelf models. 

We have similar and other relevant arguments in [General Response - Practicality and Impact]. 

For our discussion on vision language models like CLIP, please see our [General Response - Have We Evaluated Strong Methods?].

Reviewer comment (Experiments):
How will the system evolve if vision language based models like CLIP need to be tested - as that is more practical?

Author response:
> The use cases and applicability in real-life scenarios like robotics is not well established, requesting to look to the plethora of work in embodied intelligence and adapting the problem in that regard.
> How fruitful are game environments to real life human occupied or industrial environments? Is the transfer easily equitable? 
> How will the system evolve if vision language based models like CLIP need to be tested - as that is more practical? 

There is a lot of interest in the robotics community in using language as a medium decomposing tasks into a sequence of subtasks that better match the primitives available to robots. We see mapping and navigation as one such example. Indeed, our results may not be immediately applicable to real-world embodied agents. However, as people in the robotics community increasingly rely on available LLMs for reasoning, there is merit in understanding the capabilities of the off-the-shelf models. 

We have similar and other relevant arguments in [General Response - Practicality and Impact]. 

For our discussion on vision language models like CLIP, please see our [General Response - Have We Evaluated Strong Methods?].

Reviewer comment (Experiments):
How fruitfull are game environments to real life human occupied or indutstrial environments? Is the transfer easily equitable?

Author response:
> The use cases and applicability in real-life scenarios like robotics is not well established, requesting to look to the plethora of work in embodied intelligence and adapting the problem in that regard.
> How fruitful are game environments to real life human occupied or industrial environments? Is the transfer easily equitable? 
> How will the system evolve if vision language based models like CLIP need to be tested - as that is more practical? 

There is a lot of interest in the robotics community in using language as a medium decomposing tasks into a sequence of subtasks that better match the primitives available to robots. We see mapping and navigation as one such example. Indeed, our results may not be immediately applicable to real-world embodied agents. However, as people in the robotics community increasingly rely on available LLMs for reasoning, there is merit in understanding the capabilities of the off-the-shelf models. 

We have similar and other relevant arguments in [General Response - Practicality and Impact]. 

For our discussion on vision language models like CLIP, please see our [General Response - Have We Evaluated Strong Methods?].

Reviewer comment (Evaluation):
How are the easy and hardness of the DF, RF questions come up to? How does it vary with dataset characteristic changes?

Author response:
> How are the easy and hardness of the DF, RF questions come up to?
> How does it vary with dataset characteristic changes? 

Please find the definitions of "easy" and "hard"in Sec 2.2. They depend on "edges that are not covered in the prefix" of the walkthrough. Sec 2.2 also explains how an easy question may become hard once the walkthrough gets longer, as well as how to determine if it is easy or hard based on the current walkthrough (see the definition of EASY label).

Reviewer comment (Evaluation):
What are the runtimes for the experimental evaluations?

Author response:
> What are the runtimes for the experimental evaluations? 

For GPT-3.5 and GPT-4, the runtimes are primarily constrained by the speed limitation of OpenAI API calls. Completing both tasks for all mazes on two 48GB A6000 GPUs takes: 
- 2 days for the series of Llama models; 
- 3 days for RWKV. RWKV is slower because it reads longer walkthroughs. 

We will add these details to the camera-ready version.

Reviewer comment (Reproducibility):
Are any subset minival of the dataset available for checking the model performance quickly?

Author response:
> Are any subset minival of the dataset available for checking the model performance quickly?

We will publish all the data and our experiment logs once the paper is published.

Reviewer comment (Experiments):
Also, why restriction to GPT based models only?

Author response:
> Also, why restriction to GPT based models only? 

By "GPT based", do you mean "Transformer" or "decoder-only" or "OpenAI GPT"? 

Decoder-only models become predominant because they can generate texts and they are efficient to train and do inference with. Encoder-only models can not generate; encoder-decoder models are not efficient. 

We evaluate the strongest-to-date LLMs and they are all decoder-only. 

Most of them are Transformer-based. RWKV is an RNN-Transformer hybrid. 

Only two models are OpenAI GPTs. We also evaluated Llamas and RWKV.

Reviewer comment (Theory):
How is ambiguity in location and maps resolved? Any technical relation with length of text description?

Author response:
> How is ambiguity in location and maps resolved? 

A.2 and A.3 are about location and move resolutions.