Paper: Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading (ICLR-2024)

Reviewer comment (Experiments):
It could be useful for the reader if you could provide more empirical results with other LLMs, e.g., GPT3.5, GPT4, Llama 2 etc.

Author response:
We provide comparison to LLaMA 2 chat of size 13B and 70B in Table 2 and 3. We are happy to provide comparisons on GPT-4 and Claude, but we do caution that due to opacity of these systems, the results might suffer from issues like data contamination. We are happy to provide extra experiments on these models for reference in the updated version.

Reviewer comment (Evaluation):
No error bars for results. Could you include error bars for the results in Table 2, Table 3, Figure 2, Figure 3, Table 4, Figure 4 and Figure 5. Given there are no error bars, the results seem marginal, particularly for Table 2, and it appears the method is only performing significantly well on the GovReport task. Including error bars would help the reader see which results are statistically significant and not overlapping.

Author response:
We will provide significance test statistics in the updated version.

Reviewer comment (Experiments):
For Contriever (the retrieval baseline), it would be more indicative if you only included the top k retrieved responses, where $k=3$ for example or even less, instead of including all segments until they fill the context (as this could bloat the context with irrelevant information---potentially leading to worse performance).

Author response:
We provide the new retrieval baselines results as suggested on QuALITY:
| k = 1 | k = 2 | k = 3 |
|-----------------|-----------------|-----------------|
|  32.6 |  41.1 |  54.0 |



Results show a consistent trend that increasing the number of retrieved chunks improves performance. The issue of distraction by irrelevant content has not been present in our setting. We added the suggested experiments in the Appendix.

Reviewer comment (Evaluation):
*"If the LLM fails to generate parsable output three consecutive times, the navigation terminates and returns “no answer”."* How many times does this occur in practice?

Author response:
We provide finish ratios on each dataset where the model is unable to parse its response 3 consecutive times. On all datasets, finish ratios are above 90% and are not the major source of error in MemWalker. We added the experiments to the Appendix.

Reviewer comment (Experiments):
Can the method handle a larger maximum number of nodes $M_t$? Can you perform an ablation, perhaps showing it handling $M_t = \{ 16, 32, 64, 128\}$?

Author response:
The maximum number of nodes is restricted by the maximum allowable context length. For example, if each node has text of length X tokens, and the maximum context window for input from the LLM is C, the maximal number of nodes that can be connected to a parent is min’(M_t, C / X). This means that even if we set a high M_t value, it will be bottlenecked by the LLM context length. Setting M_t to 16 reaches the limit given the average summary length X per node, therefore increasing M_t wouldn’t change the tree structure.

Reviewer comment (Evaluation):
In Table 4, the stray ratio is quite high, is there any way to lower it? Does this indicate that the method is not the most efficient approach?

Author response:
We expect the stray ratio to improve as the reasoning capability improves. This can be achieved by better zeroshot prompting techniques, fine-tuning on navigation paths, or simply deploying a stronger LLM.

Reviewer comment (Experiments):
The results would be more convincing if they were compared against the SOTA retrieval, recurrence and length-extention tuned models. Currently, all results (except length-exntention fine-tuning) are based on Beluga 2 which is a the main weakness of this paper.

Author response:
**Comparison to SoTA recurrence & retrieval methods**

We aim to construct baselines by sharing as many comparable components as possible to our main method. Therefore, the recurrence baseline uses Stable Beluga 70B in a similar zero-shot prompting fashion despite other recurrent methods available that train on extra datasets. We chose Contriever as it was one of the best retrieval models at the time of development of MemWalker. Similarly, the final prediction is performed by the same Stable Beluga 70B model to keep components comparable.