Paper: Syntactic Representations Enable Interpretable Hierarchical Word Vectors (ICLR-2024)

Reviewer comment (Novelty):
The paper does not address contemporary interests in NLP, as it focuses on a contextless unigram embedding system tested with outdated benchmarks that haven't been used for years.

Author response:
We do not suggest using unigram embeddings but suggest a need for search of an alternative to the status-quo. The current state has made NLP non-efficient and hard to access. We have proposed a method that can present a new dimension by adding isolated regularities to transform the embeddings as required.

Reviewer comment (Reproducibility):
There is missing detail about implementation, such as insufficient explanation of the mentioned normalization process and lack of explanation on how statistical significance in Table 2 is determined.

Author response:
We have added the details to the normalization process, in the Appendix A1 section. [...] The new version of our work includes addition of statistical significance test, new baseline model, tests per word size, singularity value tests and much more. We have also updated the title to suggest that syntactic representations are the interpretable ones.

Reviewer comment (Novelty):
The authors failed to justify why anyone should use contextless unigram embeddings when contextual embeddings perform much better for all applications currently of interest.

Author response:
We do not suggest using unigram embeddings but suggest a need for search of an alternative to the status-quo. The current state has made NLP non-efficient and hard to access. We have proposed a method that can present a new dimension by adding isolated regularities to transform the embeddings as required.

Reviewer comment (Theory):
The authors failed to account for polysemy, which is an unrecoverable problem because many words can take on different parts of speech depending on context. For example, "read" could be a noun or verb depending on context. This is a fundamental flaw in any syntactic encoding system that does not account for context.

Author response:
Regarding polysemy, that is a major limitation of our work. Although, our work cannot distinguish senses, it does not conflate any wore to one sense. It raises the potential of being able to add context as an isolated regularity on top of contextless embeddings.

Reviewer comment (Evaluation):
The interpretability results are questionable because the number of classes is very small, and there are far more parts of speech than those included. The omission of categories such as prepositions or determiners highlights the limitations of the approach.

Author response:
Regarding the interpretability issue, we were limited to 4 classes as the oracle, WordNet is also limited to the same classes. However we have presented a newer, more detailed classification report and a full eight class classification in Appendix Word Classification Google, where we show the low number of words in other classes impact the entire classification. Also, as we wanted to present a better visualization of Interpretable embeddings, we went forward with eight parts of speeches as it is easily understandable.

Our work is more interpretable in a visual sense but lacks to the accuracy of probing. We provide a sort of "looking through the models eye" in terms of interpretability, so eight parts of speech was the considered better path forward.

Reviewer comment (Evaluation):
No uncertainty/confidence/error bars on experimental results, or significance testing.

Author response:
Thank you for the review. We have taken the suggested weaknesses and improved our work by adding statistical significance test, new baseline model, tests per word size, singularity value tests and much more. We have changed the title to signify that Syntactic Representations are the ones that are interpretable.

Reviewer comment (Experiments):
The experimental results were compared against a simple baseline thus the original embedding. It never showed how it compared against existing baselines.

Author response:
Thank you for the review. We have taken the suggested weaknesses and improved our work by adding statistical significance test, new baseline model, tests per word size, singularity value tests and much more. We have changed the title to signify that Syntactic Representations are the ones that are interpretable.

Reviewer comment (Experiments):
How sensitive is your proposed method to the size of the word list used to compute the eight parts of speech directions? Providing a similar plot shown in Appendix I of https://openreview.net/pdf?id=TkQ1sxd9P4 should be enough. You could check its sensitivity on an intrinsic or extrinsic task.

Author response:
As suggested, we have performed the size test to test sensitivity to size, where we have observed that the performance of Syntactic Representation is dependent on size but that of hierarchical vector is not. We have attributed this to the pre-trained vectors heavier influence on the hierarchical vector. The syntactic representations were sensitive to size to an extent but after the point of 1000 words, due to having to impute many words as WordNet only has four classes, the sensitivity fell.

Reviewer comment (Evaluation):
Glove and Word2vec have been shown to have some inherent structural profile with most of the words being clustered along the long principal component (https://arxiv.org/pdf/1702.01417.pdf). After applying the proposed postprocessing technique could you measure how the structure of the space changes by providing a before and after number of the largest singular value?

Author response:
We have performed the suggested structural test by providing the value of largest singularity and have observed that transformation decreases the value suggesting a potential decrease in noise and semantic nuance. We have presented this in the updated paper. This experiment has shown that we can increase a specific potential of the embedding while keeping the overall performance consistent. As a new dimension, we believe that this method could possibly be replicated to add context to contextless embeddings.

Reviewer comment (Theory):
Does your proposed method enforce an orthogonality between the new basis vectors?

Author response:
No, we have not ensured orthogonality in the bases.