From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper38/Authors
title: Re: Reviewer1
comment: Thank you for your helpful review and suggestions!

Regarding your first suggestion, we agree that a metric to measure the deviation from gaussian would be very useful. However, at this point, the null model is not as well defined as in ordinary Random Matrix Theory. Comparing the moments, (e.g. Kurtosis test) can perhaps somewhat show what we want. We will investigate this further and try to include this in final version or future works.

Furthermore, regarding the second suggestion, thank you for pointing this out, and we will try to include these in the final version.

Finally, regarding code, most of the code is using slightly modified versions of pre-existing code bases, so we can definitely share the code once the anonymity period is over.

Thank you.

From: NIPS.cc/2018/Workshop/IRASL/Paper38/AnonReviewer1
title: A sound approach for a deep problem
review: The paper presents a column analysis of a word embedding matrix in order to asses the semantic coherence of human-interpretable features in the latent space. This analysis is performed through eigenvector analysis of the SVD of the PPMI matrix computed on word co-occurrences. 

The interpretability is quantitatively assessed evaluating the distribution of the eigenvectors with respect to the normal one and through the Inverse Partecipation Ratio, that describes the structural sparsity of the eigenvectors. Moreover, qualitative analysis is performed. Word embeddings trained using the skip-gram model are used as a baseline. 

I think that the evaluation is sound and gives a strong evidence of the hypothesis that is under investigation. Though, to make it even more robust, I would suggest of:
1. compute some overall metrics for the dispersion of the distribution of the eigenvectors w.r.t. the normal one;
2. compute some overall metrics for the inverse partecipation ratio: average, standard deviation, max and min;
Moreover, for the publication, I also suggest to make the code used for the evaluation accessible in order to make the results more reproducible.

Said that, I don't see any issue that would prevent the paper from being accepted.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper38/Authors
title: Re: Reviewer2
comment: Thank you for your critical review and comments.

We agree that automatic meaning extraction from the columns will be a very interesting and practically useful direction.

We also agree that comparison with transformed word embeddings as in [1] or [2] is also certainly useful in determining which method is more superior. However, the methods employed in [1] and [2] to show interpretability is through word intrusion testing, which often requires human evaluation. One possible way would be to use the automatic evaluation used in [1] for comparing with both [1] and [2], although not as accurate as human evaluation. 

Both of your comments are duly noted for further improvements of this work.

Thank you.

References:
[1] Sungjoon Park, JinYeong Bak, and Alice Oh. Rotated word vector representations and their interpretability. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,

[2] Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. Sparse overcomplete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing

From: NIPS.cc/2018/Workshop/IRASL/Paper38/Authors
title: Re: Reviewer3
comment: Thank you for your thorough review and helpful comments. We also believe that carrying out similar analysis on pre-trained language models (i.e. BERT and ELMo) would be very interesting and useful to see, and plan to include them in future work.

Regarding your question on \mu_{u^k} and \sigma_{u^k} for the "Distribution of Eigenvector", \mu_{u^k} and \sigma_{u^k} are the mean and standard deviations of the components of the eignevector u^k. For example, if u^k is a 3 dimensional vector of [1.0, 2.0, 3.0], \mu_{u^k} and \sigma_{u^k} will each be 2.0 and 1.0.

From: NIPS.cc/2018/Workshop/IRASL/Paper38/AnonReviewer2
title: A solid and insightful look at the interpretability aspect of word embeddings
review: The manuscript seeks a word embedding representation whose column space can easily be interpreted into semantically coherent groups. This allows individual word vector dimensions to be viewed as human-interpretable semantic features. The embedding is obtained via the left eigenvectors of the SVD of the PPMI matrix of word co-occurrences. Interpretability is measured through distributions of eigenvector elements, structural sparsity of eigenvectors, as well as qualitative assessment, and compared to a baseline skip-gram embedding.

- Pro's:

The paper is well written and interpreting the dimensions of word vectors obtained through the proposed method as distinct semantic groups/latent topics is intriguing. (Perhaps the most significant finding is that skip-grams, unfortunately, do not exhibit a similar structure.) In addition, the experimental results are competently set up and quite insightful for the community. 

- Con's:

I have two minor reservations:
* the actual meaning of the columns is still subject to human interpretation - it would be nice to be able to post-process the columns to somehow distill the meaning automatically
* the authors do not compare their results with methods which transform standard pre-trained word embeddings, such as [10] — this would be useful to determine whether the direct method is genuinely superior (and/or perhaps, as just mentioned, more amenable to distilling meaning automatically).

rating: 5: Top 15% of accepted papers, strong accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper38/AnonReviewer3
title: Sound motivation and well designed experiments 
review: This paper investigates word embeddings trained with SVD and attempts to search for interpretable structure in the dimensions of the obtained embeddings. Via well thought out experiments, the paper confirms the hypothesis that semantic groups form among words that have a large value in a particular column. In doing so, the paper also introduces some new methods (borrowed from Random Matrix Theory) to analyze word embeddings. It'll be useful to see if such an analysis can be carried out for more recent embedding based methods such as BERT and ELMO. 

Pros:
The problem is well motivated, and the experiments are well designed. 

Cons:
Some of the experiments are very hard to understand and the notation is not clearly defined. For example, in the "Distribution of Eigenvector" experiment, what is \mu_{u^k} and \sigma_{u^k}? 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature