Paper: Relational Convolutional Networks: A framework for learning representations of hierarchical relations (ICLR-2024)

Reviewer comment (Evaluation):
The authors claim the method to be more interpretable and parameter-efficient manner, but there is no analysis for the same.

Author response:
Thanks for the suggestion! We have added a section in the appendix analyzing the representations learned by the MD-IPR and RelConv layers on the "contains set" task. This is an appendix C of the updated pdf. The findings are quite interesting. We observe that the encoders in MD-IPR learn to encode and separate the four latent attributes in the task (number, color, fill, and shape). Moreover, the learned graphlet filters process these relations to *exactly separate* triplets of objects which form a "set" from those that do not. Please see Figure 7-9 in the appendix.

We also added a discussion of parameter efficiency in Appendix B.1. The key idea is that the number of parameters in MD-IPR and RelConv is independent of the number of objects and the number of groups considered. This leads to computational and statistical benefits, as useful transformations are shared across pairs of objects and groups of objects. That is, analogous to the translation invariance of CNNs, the inner product relations are shared across pairs of objects and the graphlet filters are shared across groups of objects.

Please take a look and let us know if this addresses your concern. We'd appreciate any further comments or questions you may have.

Reviewer comment (Reproducibility):
Some techinical details need to be claimed

Author response:
Can you clarify what you mean by this?

Reviewer comment (Presentation):
More explanations about Fig.2 should be included, like symbol defination, and how you process pooling on graph.

Author response:
Thanks for the suggestion. We have expanded the description in the caption. Please note that the notation is defined in the main body of the text, as well as in the annotations in the figure. The first row of the figure depict the graphlet filters which correspond to the parameters $\mathbf{f} \in \mathbb{R}^{s \times s \times d_r \times n_f}$. In this example, the filter size $s$ is 3. In the second row, the relation tensor $R$ is depicted as a graph with 5 nodes, and the red highlights correspond to each relation subtensor of $s=3$ objects---$g \in \mathcal{G}$ denotes a subset of $s$ objects, and $R[g]$ denotes the subtensor of $R$ corresponding to those objects (as defined in section 3.1 of the text). For each $g \in \mathcal{G}$, $R[g]$ is compared against the filter $\mathbf{f}$ via the relational inner product $\langle \cdot, \cdot \rangle_{\mathrm{rel}}$, as defined in equations 3 and 4. This yields the relational convolution $R \ast \mathbf{f}$ as defined in equation 6. Finally, given a group matrix $G \in \mathbb{R}^{m \times n_g}$, the relational convolution *given* $G$ is computed as described in equations 7-9.

Reviewer comment (Writing):
In relation convolution layer, can you introduce the movitation of the design of attention-based pooling and more clear explanation about the techinical details?

Author response:
Are you asking about the context-aware grouping of equation 11?

Here, the proposal is to generate a group matrix $G$ which assigns objects to soft groups using not only the object's own features, but also based on the context in which the objects appear. In general, this can be done via a message-passing neural network, as described in equation (11), by obtaining an embedding $E_i \gets \mathrm{MessagePassing}(x_i, \{x_1, \ldots, x_m\})$ for each object. The message-passing operation can learn to incorporate the relevant information about object $i$'s context in $E_i$ such that $E_i$ can be mapped to a vector $\mathrm{MLP}(E_i) \in \mathbb{R}^{n_g}$ encoding the object's group membership as a function of its context within the sequence. This contextual information may include information about the features of other objects in the sequence as well as their relation with object $i$.

In the experiments, we use a Transformer Encoder block to encode the context (i.e., via self-attention). This is among the additions in the updated pdf. Please see section 5.1, Figure 5, and Table 3 in the appendix.

Reviewer comment (Experiments):
In experiments, do you treat each pix of image as an token input for transformer? or a patch like ViT?

Author response:
The input to the Transformer is a sequence of $m$ vector embeddings of each object image, obtained via a CNN. The sequence of objects are $(x_1, \ldots, x_m)$ where each $x_i \in \mathbb{R}^{w \times h \times 3}$ is an RGB image. Each image is processed independently using a CNN that produces a vector embedding $z_i \in \mathbb{R}^d$. The vector embeddings $(z_1, \ldots, z_m)$ are passed as input to the Transformer, just as for the other architectures. The CNN Embedder has a common architecture and the details are described in the appendix (Table 2 and Table 5, respectively, for each set of experiments).

Reviewer comment (Evaluation):
Despite interpretability being presented as an advantage of this architecture, there is no qualitative analysis of the results on SET or relational games. It would be fun to see how the model has implicitly learned to group things according to shapes and patterns, and how the final decision comes together by aggregating all this information.

Author response:
We agree that this would be interesting to see! We added a section to the appendix (section C) exploring and visualizing the representations learned by the MD-IPR and RelConv layers on the SET task. The results are interesting---we indeed see that different encoders in the MD-IPR layer learn to extract and compare different choices of the latent attributes in the data (i.e., number, color, fill, and shape). We also find that the RelConv layer learns graphlet filters which exactly separate triplets of cards which form 'sets' from those that don't! We hope you find this added analysis interesting.

Reviewer comment (Theory):
In Table 2 of the appendix, I notice that you’ve used only a single layer of MD-IPR and RelConv before flattening and MLP. If my understanding is correct, second-order relations need **two** layers to be represented accurately by your model. How is it that you’re still able to model SET and the “match pattern” accurately?

Author response:
Good question! Your intuition is correct that each additional layer of MD-IPR+RelConv models relations of one degree higher. In the case of the "match pattern'' task, the relations are indeed second-order, but the second-order relation is relatively simple. Having extracted the first-order relation among the triplet of objects, the MLP needs only to determine whether those first-order relations are the same or different. Thus, while a second layer of MD-IPR+RelConv would work here as well, it is not necessary.

One way to think about this is that, given a sequence of objects, an MD-IPR layer computes first-order relations. Then, following a relational convolution layer, we obtain a sequence of objects, each summarizing the relational pattern within some group. Then, a second MD-IPR layer computes the relations between those relational objects, forming second-order relations. The RelConv layer in-between can be thought of as computing relations of an order that is part-way between first-order and second-order, because it has already grouped objects and computed the relational pattern between them.

Because the final second-order relation is very simple (same or different), an MLP can parse the output of the RelConv layer into the final prediction.

One noteworthy thing here is that, with the addition of learned grouping, RelConvNet can perform even better on the generalization split of the "match pattern" task in the relational games benchmark (see Figure 5 of the updated pdf where we have now added these results).

Reviewer comment (Experiments):
I can think of a much easier way to do the grouping layer described in Section 4 - just have a single linear transform of dimension $|\mathcal{G}| \times n_g$, with learnable parameters (and probably a Softmax). I assume you must have considered this and rejected it; could you give a short explanation of why? Is it because it is more parameter-efficient to use a matrix of size $m \times n_g$?

Author response:
Parameter-efficiency is a big part of the reason. $\mathcal{G}$ will typically be exponential in the number of objects (e.g., of size $\binom{n}{s}$). This may be prohibitively large (computationally) when $n$ is moderately large. Another reason is the "statistical" aspect of how easy it would be to learn reasonable groups. We think this inductive bias makes learning meaningful groups easier because it links possible groups in $\mathcal{G}$, through object membership within them, rather than considering each group independently. In the case of temporal grouping, for example, group membership is determined by the temporal (positional) order in which the object occurs. It might be much more difficult to learn this kind of pattern with a parameterization in terms of $|\mathcal{G}|$ independent discrete groups instead. The $m \times n_g$ structure of the group matrix enables natural versions of the feature-based grouping and context-aware grouping layers as well, where an object's group assignment is simply a learned function of its features (i.e., $\phi(i, x_i) \in \mathbb{R}^{n_g}$ is a vector representing the degree to which the object belongs to each group).

Reviewer comment (Evaluation):
Could you provide some insight into the interpretability of what the model learns? For example, for the SET game, does one of the dimensions correspond to “colour”, one to “shape”, one to “number”?

Author response:
Yes! Please see Appendix C of the updated pdf.

Reviewer comment (Experiments):
Limited analysis of how the architecture scales as the number of objects and relations grow large. Memory and computation costs need investigation. The MD-IPR module computes an $n \times n$ relation tensor for $n$ objects. This grows quadratically with $n$, so could become prohibitive for large $n$. The paper does not discuss optimizations like sparsity. In relational convolutions, naively considering all possible groups of objects scales exponentially. The grouping mechanisms help, but analysis of their scaling is lacking. Stacking many relational convolution layers could substantially grow the sequence length, increasing memory and computation per layer. The paper does not report metrics like parameter count or floating point operations. There is no experiment systematically increasing the number of objects and relations to analyze how performance degrades and costs increase. The experiments use fairly small input sizes.

Author response:
Thank you for the constructive suggestion! Indeed, it is possible to reduce the computation of the $n \times n$ relation tensor. Since the relational convolution considers relations *within a group*, we only need to compute relations between objects which co-occur in the same group. We have added an appendix section (B) which discusses how this can be done. Please also see the slightly-updated discussion on computing group-match scores in section 3.2. The key quantity is the set of discrete groups and group match scores, since a sparse relation tensor can be computed based on the "support" of those quantities. That is, $\mathcal{R} = \\{R_{ij} : \exists\, g \in \mathcal{G} \ \text{such that} \ i,j \in g\\}$ in the case of convolutions with fixed discrete groups, and $\mathcal{R} = \\{R_{ij} : \exists\, k \in [n_g], g \in \mathcal{G} \ \text{such that} \ \alpha_{gk} > 0 \ \text{and} \ i,j \in g\\}$ in the case of learned soft groups. By choosing a sparse normalizer in equation (7), $\alpha_{gk}$ will be sparse and hence $\mathcal{R}$ is sparse. For example sparsemax is used in the results added to the relational games experiments in section 5.1.

Reviewer comment (Experiments):
Main experiments are on simple synthetic tasks. The relational games and SET tasks have at most simple second-order relations. Testing on tasks requiring modeling higher-order relations would better evaluate the capabilities of the relational convolutional networks. Besides, these tasks have a small fixed number of objects. Scaling to variable, larger numbers of objects would be more realistic.

Author response:
We agree that more complex tasks, requiring modeling higher-order relations, would form an interesting evaluation of the architecture. Since existing relational architectures (e.g., PrediNet, CoRelNet, RelationalNet) are non-compositional, and did not focus on learning hierarchical relations, we were unable to find benchmarks for explicitly higher-order relational representation. The SET experiment is a step towards this, but we agree that evaluation on even higher-order relations as well as more realistic tasks would be interesting. The construction of such benchmarks requires careful thought and consideration, and was outside the scope of this project. However, we did add a section to the appendix (section D) to propose some early ideas on tasks involving higher-order relations and potential real-world applications of the relational convolutions architecture.