Paper: QORA: Zero-Shot Transfer via Interpretable Object-Relational Model Learning (ICLR-2024)

Reviewer comment (Presentation):
The proposed model is difficult to conceptualize, especially without a supporting figure.

Author response:
We have added a figure to the Appendix, with a reference in Section 3.2, to help the reader
understand the overall operation of our algorithm.

Reviewer comment (Experiments):
The proposed benchmarks, while suitable as toy tests, do not extend to nuanced representation learning and remain within grid worlds, where performance does not always translate to more realistic settings.

Author response:
QORA is not limited to grid worlds; instead, it can
operate in a wide variety of scenarios that follow causality
principles that can be conceptualized by predicate logic. Hence, the
environments included in the paper should be viewed as examples that
demonstrate specific aspects of QORA's functionality rather than its
inherent limitations. To highlight this further, the revised paper
includes experiments with a new domain where QORA has to learn the
operation of a switch that toggles a number of light bulbs. As given
in the new conclusion, future work will extend QORA to additional
modes of operation (e.g., planning, exploration) and explore
domains with more complex object interactions. We consider these
topics beyond the scope of the current paper.

Reviewer comment (Theory):
There are technical imprecisions; contrary to the authors' claims, CNNs can support variable length inputs, and architectures using 3x3 convolutions can model longer-term dependencies through mechanisms like average pooling, self-attention, max pooling, and CNN cascades.

Author response:
Indeed, we could've worded this better. The intent was to communicate that CNNs would not work (at least, not in a reasonable way) with our object-based state representation. In the revision, the relevant text has been removed due to restructuring, so the problem is no longer present.

Reviewer comment (Reproducibility):
The neural network details used for comparison are not provided, which is a critical omission for reproducibility and transparency.

Author response:
This information eventually could be obtained
from the release of our neural-network test scripts, along with a
QORA implementation. But in the mean time, the revision contains an
Appendix with information on the neural networks used in the
comparison.

Reviewer comment (Presentation):
Figures intended for side-by-side comparison have differing scales, which could mislead the interpretation of the results.

Author response:
It is unclear which plots the reviewer refers to and/or which
axes. If this is the y-axis in the six graphs of Figure 2, there is
absolutely no way that error curves that peak at 0.4 can be on the
same scale as those that reach 100. This would make the former
basically a flat line at zero. Instead, the figures are scaled to
make the various curves clearly visible. If the question refers to
Figure 3 or 4, the various plots therein are either on the same scale
or they show completely different metrics that are not meant to be
compared (e.g., space complexity of QORA vs error).

Reviewer comment (Evaluation):
The evaluation is not sufficiently robust and lacks the depth needed to substantiate the claims made.

Author response:
We believe that our experiments are quite thorough,
especially with the new results included in the revision (e.g.,
zero-shot transfer to 128x128 worlds, the light bulb domain). If the
reviewer has the specifics about which parts of the evaluation are
not robust and/or which claims are unsubstantiated, we would be
interested in addressing them in the camera-ready version.

Reviewer comment (Presentation):
Including a figure to visualize QORA's architecture may aid in understanding. Could such a figure be provided?

Author response:
Yes, Figure 4 in the appendix visualizes some of the structures described in Section 3.2.

Reviewer comment (Experiments):
The benchmarks focus on grid-world environments. Can you extend these to more complex settings to better illustrate QORA's generalization?

Author response:
The fact that the included environments are primarily
grid-worlds is not due to a limitation of QORA; in fact, the revision
includes a new light-bulb environment that is not a grid-world (it
actually has no specific positional information at all). Rather,
grid-worlds are a convenient tool for helping the reader understand
the workings of the environment. QORA can learn rules in a fairly
general form, but many of these domains would make little sense to
humans (e.g., relations in a 7-dimensional space) or be unnecessarily
complex (e.g., rules with hundreds of predicates). Instead, the paper
urges the reader to focus on the essence of QORA's generalization
capability: QORA learns rules *as simple as possible* to better
reflect the underlying transition dynamics of the environment
*without* relying on information specific to some arbitrary
subset of states. This property holds no matter what environment QORA
is applied to, while the same universally does *not* hold for
prior work (as demonstrated by Figure 2b in the revised paper).

Reviewer comment (Experiments):
A detailed description of the neural network baselines would be beneficial. Could you provide this additional context?

Author response:
Please see response to weakness #5 (W5).

Reviewer comment (Presentation):
The varying scales in comparative figures may lead to misinterpretation. Can you ensure uniform scales across all relevant figures for clarity?

Author response:
Please see response to weakness #6 (W6).

Reviewer comment (Writing):
In the introduction, the terms "generalization" (the model's ability to make effective decisions when exposed to novel inputs), "interpretability" (how easily its learned parameters can be inspected and understood by a human), and "robustness" (the predictability of its behavior on arbitrary inputs) need clarification. Explain the distinction between "novel" inputs (previously unseen) and "arbitrary" inputs (inputs that may not follow a specific pattern or structure) to avoid confusion.

Author response:
The revised introduction no longer uses terms "novel input" or "arbitrary input."

Reviewer comment (Writing):
On page 4, when discussing the player's ability to swap its color with the "new change-color action," it's important to introduce the "new change-color action" before using it in the sentence. Provide a brief explanation or definition of this action to ensure readers understand its context and purpose.

Author response:
It is a bit unclear what the reviewer means because "changing color" is not an ambiguous action. From the rest of the paragraph in the paper, it should be clear that a) colors are binary; b) players can flip their color to the opposite using this action; and c) the purpose of changing the color is to be able to pass through doors that have the same color. In any case, we have rephrased this sentence to flow better, but feel that the rest of the paragraph is pretty self-explanatory.

Reviewer comment (Writing):
On page 5, when referring to the "best" candidate, explicitly define what "best" means within the context of the paper. Clarify whether "best" refers to candidates that are most relevant for prediction, those with the highest confidence scores, or some other criteria.

Author response:
Sentence rephrased.

Reviewer comment (Writing):
Regarding the explanation of boosting and the working set in the sentence on page 5, consider whether this explanation is necessary. If it adds value to the reader's understanding of how the working set is updated, provide a concise summary of how boosting is related to the working set, ensuring that it enhances clarity.

Author response:
Yes, boosting is an essential part of QORA, which is a method for iteratively improving a model by focusing on its current weaknesses. The paper explains this in fairly unambiguous terms: "candidates in the working set do not receive observations that the best hypothesis predicts with high confidence, thereby magnifying the score of candidates that are well-suited to correct existing errors." Due to limited space and the fact that boosting is a well-known concept in the field of ML, we do not believe that elaborating on the rationale for using this method is warranted. Instead, the revision now cites the paper that invented boosting, while as a brief reference, the reviewer is directed to the following page for more information:  

https://en.wikipedia.org/wiki/Boosting_(machine_learning)

Reviewer comment (Writing):
In section 3.2, provide a clear definition or explanation of the "learnable module" within the context of Quantified Function Learning. Specify its purpose and functionality to ensure the reader understands its role in the paper.

Author response:
The paper does not contain the phrase "learnable module", which makes it difficult for us to specify its purpose and functionality in the context of QORA. If the reviewer uses this term to refer to deep learning, where modules are composed to produce complex neural-network architectures, this concept is not relevant to the current paper. Instead, we are proposing an entirely new (non-neural-network based) algorithm for learning predictive models, where QORA (and, more specifically, the topics covered in the Quantified Function Learning subsection) are not meant to be used as composable modules in this sense.

Reviewer comment (Writing):
Equation 2 needs clarification regarding the conditions for movement to the right and the meaning of first and second coordinates.

Author response:
It is a bit unclear what the reviewer means, but generally speaking most fields agree on a convention that a position in 2D space is commonly described by a tuple in which the $x$ coordinate appears first and the $y$ coordinate appears second. Hence, movement to the right corresponds to an increase in the $x$ coordinate, which the paper confirms just above (2) by stating "a player $x$ can move to the right: $P(x): \exists y \in walls: y.pos - x.pos = (1, 0)$". In more detail, this predicate is true when there is a wall on the right side of the player's current position; hence, the move is successful iff the predicate is false. More broadly, the operation of QORA does not depend on the direction in which left/right/up/down actions move the player. Its algorithms would work exactly the same if the domain were flipped sideways, e.g., moving up would cause the $x$ coordinate to reduce.  

Since the original formulas used $x$, $y$, $z$ to prepresent objects, which may be causing confusion with coordinates, we have changed these variables to $o_1$, $o_2$, $o_3$ in the revision.

Reviewer comment (Writing):
The meaning of "c" is not provided in Section 3.3.

Author response:
It appears the reviewer missed that $(c, m, a)$ triplets were previously introduced at the beginning of Section 3, where the paper states "We separate transition rules based on the (class, member attribute type, action) $(c, m, a)$ triplet they apply to and predict changes independently for each attribute of every object." In the revision, we have added text to Section 3.3 to remind the reader of this previous definition.

Reviewer comment (Writing):
In Section 4, different notations for different meanings should be used to avoid reader confusion.

Author response:
It is unclear what notation the reviewer refers to. Any variables that were reused (e.g., $m$) were in clearly different contexts, but to ensure there could be no notation-related confusion throughout the paper, we have gone through and changed some of the variables (e.g., the $m$ used in Section 4 is now $n$).

Reviewer comment (Writing):
The abbreviation "EMD" is used before it is introduced.

Author response:
Thank you for catching this; we have fixed it.

Reviewer comment (Experiments):
The use of \(m=1\) for Neural Networks and larger \(m\) for DOORMAX and QORA should be justified more, as it may raise fairness concerns.

Author response:
It appears the reviewer got this backwards -- as mentioned in Section 4.1 "neural networks used $m=1$ to increase learning speed." Additionally, as shown in the original Figure 2(c), decreasing $m$ leads to better performance, which means that neural networks obtain the maximum advantage over QORA when running with $m=1$. This happens because lower $m$ means the algorithm receives more variety in the training data. The correct interpretation of the results in Section 4 should be: even though neural networks run with the best possible $m$, they still cannot come anywhere near QORA's efficiency and ability to learn. To prevent this confusion in the future, we have added a new Figure 2(a) in the revision that tests the neural network baselines with varied episode lengths (now called $n$ instead of $m$). This new plot perhaps makes it more clear that having $m=1$, i.e., generating a new level for every observation, leads to best performance.

Reviewer comment (Evaluation):
The explanation for DOORMAX's inability to resolve the effect of the change-color action is insufficient.

Author response:
Although the revision no longer compares against DOORMAX or points out its inability to work with the doors domain, we can elaborate on this topic in the response. DOORMAX can learn effects of two types: arithmetic (add or subtract a constant) and assignment (set to a constant value). As part of the learning process, it attempts to determine which of these two types of effects is actually being enacted by a given rule. Notably, for a given transition, DOORMAX will not make *any* prediction if there is still a rule that has ambiguous effects. In the case of the doors domain, the effect toggles an attribute from $0 \to 1$ or $1 \to 0$, which could be considered either arithmetic *or* assignment (since the initial value of the attribute is given as input to the rule). Thus, DOORMAX is unable to disambiguate the effect type, which forces it to give up and make no prediction.

Reviewer comment (Evaluation):
The evaluation is limited to a synthetic environment, and the paper should discuss the potential for extending the model to more realistic environments and the expected limitations in such cases.

Author response:
Many reinforcement learning experiments are done in synthetic environments due to the high cost of real-world experiments (especially when millions of training iterations are required, as with most deep-learning approaches). In this case, the environments we introduce are designed to test specific aspects of QORA's learning without adding unnecessary complexity. In the revised introduction, we mention that future work will expand QORA's capabilities and address more challenging domains.

Reviewer comment (Experiments):
The choice of baselines lacks diversity, and the paper could benefit from considering relevant studies in the object-centric representation field [3,4,5].

Author response:
We do not believe the listed papers [3, 4, 5] are relevant to our problem as they mostly deal with learning object-centric representations. Instead, the current paper assumes such a representation is given, but the goal is to learn object-based state transition models of a given world. The revised introduction should explain this better, in addition to pointing out that the field lacks baselines that fit our requirements (i.e., model-based, object-oriented domain learning with no a-prior knowledge). While we found another such technique (MHDPA) and included comparison against it in the revision, the outcome is not promising, i.e., it performs as poorly as NPE.

Reviewer comment (Experiments):
While QORA tacles more challenging environments than prior works (e.g., DOORMAX_D in Marom and Rosman, 2018), the evaluations are still on small-scale games, which limits its wide applicability.

Author response:
> While QORA tacles more challenging  environments than prior works (e.g., DOORMAX\_D in Marom and Rosman, 2018), the evaluations are still on small-scale games, which limits its wide applicability.

We appreciate the comment, but QORA is applicable to a
much larger variety of environments than just those shown in the
paper, both in terms of their size (i.e., number of objects) and
the types of rules that govern their behavior. Furthermore, it should be
noted that even the domains that appear simple/small require
construction and consideration of a large number of hypotheses. For
example, the 8x8 walls domain contains 28 external (perimeter) walls
and 10 interior walls, whose semantics QORA isn't given (e.g., it
has no idea that walls cannot move). Therefore, it must consider all
possible interactions between them, as well as the position of the
agent and action taken, to build a predictive model. Given the
limited space in the paper, we focus on the most basic examples to
demonstrate QORA's particular capabilities rather than achieve an
exhaustive coverage of all possible worlds it can operate in. Future
work will focus on extending QORA to handle game playing, exploration, and planning, but
these topics are clearly beyond the scope of the current paper.

With this in mind, the revised paper contains new experiments with
larger (64x64 and 128x128) worlds, a new domain with light bulbs, and
additional discussion emphasizing some of the points in the preceding
paragraph.

Reviewer comment (Theory):
There are no theoretical guarantees provided for convergence or sample complexity.

Author response:
> There are no theoretical guarantees provided for convergence or sample complexity.

Indeed, QORA tackles much more general problems than prior
work, making theoretical analysis correspondingly more difficult and
therefore beyond the scope of this introductory paper. However, we
agree this is an important topic, which we plan to address in future
publications.