From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper46/Authors
title: Response to AnonReviewer3
comment: First, we thank the reviewer for their helpful feedback and comments regarding the article.

We agree with the reviewer that a focus of our work is in formulating a low rank matrix decomposition problem. Part of this is casting existing methods into the generalized low rank model framework by finding appropriate underling distributions. We note valuable prior works in this direction, including Cotterell (2017), Landgraf and Bellay (2017), and Li et al. (2015), which introduce (explicitly or otherwise) the multinomial-PCA framework for the Skip-gram model.  In contrast, we note that there is a shortage of valuable discussion for GloVe. (We believe we have done an exhaustive literature review, and have been unable to find more work in this direction than is cited.) We attempt to close this discrepancy in level of understanding between Skip-gram and GloVe by introducing the Tweedie generalized low rank model to derive GloVe as a one-step estimator (with algorithmic regularization via weight truncation.)

There are mainly two choices that need to made in our method: the co-occurrence matrix and distribution. We view the co-occurrence matrix as being an extraction of linguistic content from the corpus, and therefore view the choice of co-occurrence matrix as being a linguistic one beyond the scope of our work. In contrast, we view the choice of distribution as being a modelling choice which is directly a focus of our work. Two examples are given to illustrate how to find an appropriate distribution: one for each of the co-occurrence matrices existing in literature. (To our knowledge, there are no others besides the “count” and the “harmonic” co-occurrence matrices.) Along the way, connections are made to other methods in the literature. We now recognize this was not as clear as it should have been in the submission and have improved the clarity.

We agree that the reported improvement of our MLE of the Tweedie GLRM on the word analogy task is marginal compared to GloVe. We plan in future work to improve this performance. We view this as being in part a computational relic so are working on improving the efficiency of our code to do this. Another interesting direction is choosing a model to better enable large co-occurences.

Thank you for pointing out the typo: since the objective function for Skip-gram is the log-likelihood of a multinomial distribution, maximization (not minimization) indeed should be occurring.

From: NIPS.cc/2018/Workshop/IRASL/Paper46/Authors
title: Response to AnonReviewer2
comment: First, we thank the reviewer for their helpful feedback and comments regarding the article. 

With regards to the suggestions, we are working on a rewrite of the appendix and background sections to incorporate important definitions and references into the main body of the paper while additionally moving parts of the paper which are less important into the appendix. 

It is correct that the first iteration within the tables (specifically Table 1) is the GloVe algorithm comparison. We will clarify within the results section which results come from prior methods. We apologize for any confusion or difficulty in understanding this may have caused. 

With regards to interpretability, as was pointed out in the review "the method has clear assumptions, respect optimization problems involved and considers probability distributions." We would like to emphasize that all prior methods for word embedding mentioned in the paper such as GloVe, Skip-gram, and SVD all have implicit probability distribution assumptions, and one of the goal of this paper is to make these explicit. In our view, this connection between methods and models leads to interpretability, since questions about the method or its output can be pulled backed into questions about the model, which is well understood. (For example, when trying to interpret a least squares estimator in regression, knowing the Gaussian distribution provides significant mileage. In the GLM and GLRM setting, the same holds.) We have tried to make this clearer in the most recent version of the paper. We are not, for instance, directly interpreting the word vectors or directly explaining how the inputs conspire to influence the output.

From: NIPS.cc/2018/Workshop/IRASL/Paper46/AnonReviewer3
title: An interesting idea, but the practical and the theoretical contributions are both marginal
review: Many word embedding learning methods can be considered as a low-rank matrix decomposition problem with the target matrix becoming word cooccurrences and there are many valuable discussions on this direction in previous papers.
This paper follows the same direction, while it extends the traditional PCA method to "generalized low rank model" to form the decomposition problem, which is inspired by "generalized linear model". The benefit of such formulation is that, one can directly borrow the IRLS algorithm in generalized linear model to handle the optimization in  "generalized low rank model".
The whole idea is interesting and it can fit into some of the popular embedding approaches to some extent. While the gain from the algorithm is minor compared with standard GloVe, and the interpretation is not distinguishable enough to make itself become a totally new formulation/explanation of the problem. A much more important question (both practically and theoretically) is that, how can we design a better cooccurrence matrix Y and loss function L?

error:
The equation (1)  should be "argmax", not "argmin"?
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper46/AnonReviewer2
title: Good paper, both in theoretical and experimental aspects. 
review: In this paper, authors present a methodology for computing word vectors from a corpus.
The method has clear assumptions, respect the optimization problems involved and the probability distributions considered. This approach “generalizes” two previous method used to compute word vectors. Experimental results show an improvement respect other methods, and in general, the paper is well written. 

Overall, it is a good paper. Just a few comment that I consider to improve:
1) Improve the references to the appendix. Throughout the paper there are many references to definitions and results presented in the appendix. This don’t help to the fluency of the paper and sometimes is kind of confusing.
2) Clarify experimental results. In the tables is not clear the comparison with other methods. We deduce that GloVe appears as results of the first iteration of the proposed method. Indicating this directly on the tables can help the reader.
3) It is not clear the benefit in interpretation. In fact, despite the title suggests that the interpretation of the word embeddings is part of the paper, there is no result related to that. It can be helpful to clarify this.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct