Paper: Bridging Neural and Symbolic Representations with Transitional Dictionary Learning (ICLR-2024)

Reviewer comment (Presentation):
The major weakness of the paper is the bad presentation of Sec. 3&4 that greatly hinders the readers from understanding the paper.

Author response:
We have substantially revised Sections 3 and 4 for better clarity and coherence, with changes highlighted in blue in the revised manuscript. These revisions are intended to address the issues you raised and improve overall readability.

Reviewer comment (Writing):
The annotation in Sec. 3 is in quite a mess. Scalar value, vector, set, and matrix are not in consistent forms, and multiple critical variables lack clear definition/explanation.

Author response:
We meticulously reviewed and standardized our notation, adopting a consistent sub/super-script convention detailed at the start of Section 3: ***“Superscript $·^i$ denotes $i$-arity, superscript $·^{(i)}$ with brackets indicates the $i$-th sample in a dataset, and subscript $·_i$ refers to the $i$-th visual part in a sample.”*** This clarification aims to resolve any ambiguity in our notation.

Reviewer comment (Writing):
What does 'a' stand for in "such as Cat(a), Tree(a), Person(a)"?

Author response:
We clarified that ‘a’ represents a logical variable, revising examples to use $x_i$ (e.g., Cat($x_i$)) directly for greater clarity.

Reviewer comment (Writing):
What is the relationship between \(x_i\) and \(x\)? It seems the pieces of \(x_i\) are determined by the masks and are directly combined into a whole instead of the linear addition.

Author response:
$x_i$ is a visual part of image $x$, it is true that under our linear assumption, $x=\sum_{i=1}^{N_P} x_i=\sum_{i=1}^{N_P} m_ix$, the parts can be decided by masks, that is the reason why we introduce the game theoretic mechanisms to avoid collapsed solutions, i.e. directly output an all-1 mask.  

We rewrote paragraph 3 in Section 3.1 to introduce the linear assumption in detail first, followed by a discussion on symbol grounding, in order to reduce such ambiguity.

Reviewer comment (Writing):
What is the relationship between \(r_i\) above Eq. 1, \(R_i\) in Eq. 1, and \(r_j^i\) below Eq. 1?

Author response:
We solved this kind of ambiguity by the sub/superscript convention discussed above. $R^{(i)}$ is the transitional representation or neural logical variables (each variable is an visual part) for $i$-th sample in the dataset, $r_i$ is the representation for the i-th part in $R$, $r^{(i)}_j$ is the representation for the $j$-th part in $i$-th sample.

Reviewer comment (Theory):
How can \(\theta\) be optimized in Eq. 1 if it does not appear in the two terms? The definition of the decoder \(g(\cdot)\) is not consistent. Does it take \(\theta\) as a condition or not? It seems Eq. 2 is the appropriate form.

Author response:
Yes, the theta (dictionary) should be used in both $f$ and $g$, while $f$ and $g$ (the diffusion model) should also have their own parameters to be optimized.  

We included theta (dictionary) in both the $f$ and $g$ and provided additional clarification on how $R$ is derived from $f$ in Equation (1).

Reviewer comment (Writing):
The definitions of two critical terms \(E_{\tilde{D}}\) and \(d_S\) are unclear.

Author response:
We expanded the explanation of the expectation term and the ideal metric distance in the revised paper by rewriting paragraph 5 in Section 3.1, we introduced details and examples to clarify these critical concepts.

Reviewer comment (Reproducibility):
How is the dictionary \(\tilde{D}\) obtained given the argument "As we have meaningful \(\tilde{D}\)"?

Author response:
We fixed this ambiguous sentence by rewriting Section 3.1 paragraph 5, it is not we have meaningful dictionaries, but we only need to consider meaningful dictionaries. We also updated the corresponding sentence to ***“As we only need to consider meaningful dictionaries…”***

Reviewer comment (Reproducibility):
It seems that the only variable to be optimized is the hidden dictionary \(\theta\). What about the models of \(f(x;\theta)\), \(\hat{g}(r_i;\theta)\), \(g_{\theta}(R^i)\), and \(g_{\tilde{D}(R)}\)?

Author response:
We clarified that the parameters of the decomposition model ($f$ and $g$) are also optimized, alongside the dictionary, to resolve any confusion and added this sentence: ***“we omit the parameters of decomposition model f and g that also need to be optimized in this target for simplicity.”***.

Reviewer comment (Presentation):
The illustration of Fig. 2 does not clearly demonstrate the formulation in Sec. 3 and the solution in Sec. 4.

Author response:
Figure 2 has been updated to better align with the formulation in Section 3, including annotated mathematical notations and more details about the player models.

Reviewer comment (Presentation):
\(f, R, r_i, g, x_i, m_i\) are not clearly labeled in the figure.

Author response:
We added labels to the figure to clarify them.

Reviewer comment (Presentation):
Where is the \(N_P\) copies of the model in the figure?

Author response:
We added labels to the figure to clarify them.

Reviewer comment (Theory):
Why is there a "GT loss" in an unsupervised learning pipeline?

Author response:
GT loss is to avoid collapsed outputs as discussed above. It is unsupervised as it does not rely on any label. It introduced additional inductive biases to remove collapsed solutions from its search space.

Reviewer comment (Presentation):
Where is the "Decomposition Loss" mentioned in Fig. 2?

Author response:
Decomposition loss is written at the end of Section 4.1, we bolden it, we also renamed the Reconstruction loss to reconstruction error as it is a part of decomposition loss to avoid such confusion.

Reviewer comment (Experiments):
The authors are encouraged to provide qualitative comparison results on Line World and ShapeNet5.

Author response:
The concerns about Sections 3 and 4 are addressed above. We acknowledge the importance of providing qualitative comparisons on Line World and ShapeNet5. We will include these additional insights in the final version of our paper.

Reviewer comment (Experiments):
Experiments are limited to tiny, mostly binary datasets such as "ShapeNet5", which is basically a subset of 5 categories from the ShapeNet dataset. It is not clear if the methods would generalize to noisy real-world data, such as training using noisy, incomplete instances where the parts are not all visible.

Author response:
Our experimental setup is delicately designed to exclude visual feature confoundings. This approach substantiates our main claim that TDL can learn compositional and reusable visual predicates from high-dimensional input. Extension to noisy real-world data is recognized as an important future direction that is out of the main scope of this paper.

Reviewer comment (Experiments):
Although the paper provides a reasonably well-curated list of neuro-symbolic approaches, the evaluations do not compare against any of the recent approaches. Instead the comparison is against clustering baselines.

Author response:
The neuro-symbolic approach is a broad concept, it can cover almost all kinds of tasks while baseline picking is task-specific. In our task, unsupervised learning of concepts and relations (predicates) from visual data, the unsupervised part segmentation methods are the most relevant. To the best of our knowledge, we do not see a suitable, directly comparable neural symbolic baseline, especially for this problem.

Reviewer comment (Experiments):
The paper reads as a mishmash of several different ideas that are used together, but not integrated coherently. Therefore having ablation studies to show the value of each module would be crucial. However, the evaluations do not provide a clear understanding of the contribution of each component to the overall methodology.

Author response:
We improved our writing to make the structure of our method clear. Our method is basically an EM algorithm implemented by two components, clustering and a decomposition model (i.e. the game-theoretic diffusion model). We provided the ablation study in Appendix E, “Base” vs ”+Cluster”, the model with and without clustering, we further provided an ablation study on the commonly used optimizations for the decomposition model. 

Inside the decomposition model, its loss terms are all necessary to guarantee the right function (right convergence) of the model, the reconstruction error is the basis, and the other GT loss terms are designed to avoid collapsed outputs which are further discussed in Appendix D, SMLD loss is to train the scoring based diffusion model we used. They cannot be further decomposed into an ablation study.

Reviewer comment (Evaluation):
Ultimately, the task of reconstructing and explaining shapes simultaneously might be quite ambiguous as depicted in figure 4, and might not generalize to natural datasets. These aspects are not addressed in the paper.

Author response:
We modified the caption of Figure 4 from:
***“Comparison between our method and baselines on the OmniGlot test set. Our method can learn interpretable strokes compared to the baselines that failed to give effective strokes.”***
 to:
***“Examples from OmniGlot test set. Our method generates multiple interpretable strokes to reconstruct the input hand-written characters. As a comparison, the baseline methods segment the input into colored parts that are not valid strokes revealing a failure in learning compositionality.”***

It introduced more details about the difference between our method and baselines, and why our method can be regarded as successful in learning reusable predicates.

As discussed earlier, It is natural to extend the basic TDL proposed to natural data, it firstly requires considering the noise as discussed, and secondly the additional information, including the visual features like edges, shadows, and textures, which are normally related to the use of pre-trained backbones.

Reviewer comment (Evaluation):
Are the predicates shared among different classes? Do predicates always correspond to semantic attributes? It would help to visualize the learnt 1-any and the 2-ary predicates.

Author response:
Yes, as an unsupervised learning method, the model is unaware of “classes”.  In our experiments, the learned predicates correspond to visual parts and combinations (e.g., armrest, leg), applicable across different object types (e.g. table, chair), It is also demonstrated in our ShapeGlot transfer learning to unseen classes experiment.


We rewrote Section 3.1 paragraph 5 to reduce the confusion related to this question. To answer your question, the correspondence of predicates to semantic attributes is not guaranteed, as it is subjective like there exist different fonts to write the same character with the same meaning. The learned transitional representation (i.e. predicates) is an "average" of those fonts that have the minimal possible distance to all of them by approximating their commonality, the reusable and compositional, as demonstrated in the objective target in Section 3.2. And each font is a "concrete meaningful dictionary" mentioned in Section 3.2. Such an "average" dictionary is not necessarily highly meaningful but should be close enough as we assume the meaningful ones are compositional and reusable.

We agree that visualizing the latent predicates is helpful, right now we do not have a decoder that can directly decode the predicate as an image, we are working on it and try to update it in the final version.

Reviewer comment (Experiments):
How does the method compare to other neuro-symbolic baselines? The current set of baselines are essentially unsupervised clustering methods.

Author response:
As discussed above in Weaknesses 3, to the best of our knowledge, we cannot find a neuro-symbolic baseline for our specific task.

Reviewer comment (Writing):
It would be good to have a limitations section that discusses when this method wouldn't work. How do predicates such as left_of and larger (examples from the paper) operate in case of multi-view settings, where these terms become ambiguous.

Author response:
We've added a new section in Appendix K.1 discussing the limitations. Particularly, the model's output may be ambiguous in scenarios where the problem itself is ambiguous to humans. For example, A is on the left of B in image 1, while A is on the right in image 2. Decide whether A is on the left or right will be an undecidable problem, the model will also give an ambiguous output. Similar to Figure 8, bottom left, when the model is given an invalid combination of parts, it cannot decide a clear relation.

Relations like "larger" are different as it is related to reasoning (i.e. using knowledge/rule of perspective) which is not involved in TDL as a representation method. We added the discussion of the unsupervised rule learning in Appendix K.2 under the view of TDL.