Original: L EARNING C OVARIATE -S PECIFIC E MBEDDINGS WITH T ENSOR D ECOMPOSITIONS
Revised: L EARNING C OVARIATE -S PECIFIC E MBEDDINGS WITH T ENSOR D ECOMPOSITIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Word embedding is a useful approach to capture co-occurrence structures in a large corpus of text.
Revised: Word embedding is a useful approach to capture co-occurrence structures in a large corpus of text.

Original: In addition to the text data itself, we often have additional covariates associated with individual documents in thethe corpus—e.g. the demographic of the author, time and venue of publication, etc.—and we would like the embedding to naturally capture the information of the covariates.
Revised: In addition to the text data itself, we often have additional covariates associated with individual documents in the corpus—e.g. the demographic of the author, time and venue of publication, etc.—and we would like the embedding to naturally capture the information of the covariates.

Original: In this paper, we propose a new tensor decomposition model for word embeddings with covariates.
Revised: In this paper, we propose a new tensor decomposition model for word embeddings with covariates.

Original: Our model jointly learns a base embedding for all the words as well as a weighted diagonal transformation to model how each covariate modiﬁes the base embedding.
Revised: Our model jointly learns a base embedding for all the words as well as a weighted diagonal transformation to model how each covariate modiﬁes the base embedding.

Original: To obtain the speciﬁc embedding for a particular author or venue, for example, we can then simply multiply the base embedding by the transformation matrix associated with that time or venue.
Revised: To obtain the speciﬁc embedding for a particular author or venue, for example, we can then simply multiply the base embedding by the transformation matrix associated with that time or venue.

Original: The main advantages of our approach is data efﬁciency and interpretability of the covariate transformation matrix.
Revised: The main advantages of our approach is data efﬁciency and interpretability of the covariate transformation matrix.

Original: Our experiments demonstrate that our joint model learns substantially better embeddings conditioned on each covariate compared to the standard approach of learning a separate embedding for each covariate using only the relevant subset of data.
Revised: Our experiments demonstrate that our joint model learns substantially better embeddings conditioned on each covariate compared to the standard approach of learning a separate embedding for each covariate using only the relevant subset of data.

Original: Furthermore, our model encourages the embeddings to be “topic-aligned” in the sense that the dimensions have speciﬁc independent meanings.
Revised: Furthermore, our model encourages the embeddings to be “topic-aligned” in the sense that the dimensions have speciﬁc independent meanings.

Original: This allows our covariate-speciﬁc embeddings to be compared by topic, enabling downstream differential analysis.
Revised: This allows our covariate-speciﬁc embeddings to be compared by topic, enabling downstream differential analysis.

Original: We empirically evaluate the beneﬁts of our algorithm on several datasets, and demonstrate how it can be used to address many natural questions about the effects of covariates.
Revised: We empirically evaluate the beneﬁts of our algorithm on several datasets, and demonstrate how it can be used to address many natural questions about the effects of covariates.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The use of factorizations of co-occurrence statistics in learning low-dimensional representations of words is an area that has received a large amount of attention in recent years, perhaps best represented by how widespread algorithms such as GloVe (Pennington et al., 2014) and Word2Vec
Revised: The use of factorizations of co-occurrence statistics in learning low-dimensional representations of words is an area that has received a large amount of attention in recent years, perhaps best represented by how widespread algorithms such as GloVe (Pennington et al., 2014) and Word2Vec

Original: (Mikolov et al., 2013) are in downstream applications.
Revised: (Mikolov et al., 2013) are in downstream applications.

Original: In particular, suppose we have a set of words i ∈ [ n ] , where n is the size of the vocabulary.
Revised: In particular, suppose we have a set of words i ∈ [ n ] , where n is the size of the vocabulary.

Original: The aim is to, for a ﬁxed dimensionality d , assign a vector v i ∈
Revised: The aim is to, for a ﬁxed dimensionality d , assign a vector v i ∈

Original: R d to each word in the vocabulary in a way that preserves semantic structure.
Revised: R d to each word in the vocabulary in a way that preserves semantic structure.

Original: In many settings, we have a corpus with additional covariates on individual documents.
Revised: In many settings, we have a corpus with additional covariates on individual documents.

Original: For example, we might have news articles from both conservative and liberal-leaning publications, and using the same word embedding for all the text can lose interesting information.
Revised: For example, we might have news articles from both conservative and liberal-leaning publications, and using the same word embedding for all the text can lose interesting information.

Original: Furthermore, we suggest that there are meaningful semantic relationships that can be captured by exploiting the differences in these conditional statistics.
Revised: Furthermore, we suggest that there are meaningful semantic relationships that can be captured by exploiting the differences in these conditional statistics.

Original: To this end, we propose the following two key questions that capture the problems that our work addresses, and for each, we give a concrete motivating example of a problem in the semantic inference literature that it encompasses.
Revised: To this end, we propose the following two key questions that capture the problems that our work addresses, and for each, we give a concrete motivating example of a problem in the semantic inference literature that it encompasses.

Original: Question 1: How can we leverage conditional co-occurrence statistics to capture the effect of a covariate on word usage?
Revised: Question 1: How can we leverage conditional co-occurrence statistics to capture the effect of a covariate on word usage?

Original: For example, did William Shakespeare truly write all the works credited to him, or have there been other “ghostwriters” who have contributed to the Shakespeare canon?
Revised: For example, did William Shakespeare truly write all the works credited to him, or have there been other “ghostwriters” who have contributed to the Shakespeare canon?

Original: This is the famous Shakespeare authorship question, for which historians have proposed various candidates as the true authors of particular plays or poems
Revised: This is the famous Shakespeare authorship question, for which historians have proposed various candidates as the true authors of particular plays or poems

Original: (Hope, 1994).
Revised: (Hope, 1994).

Original: If the latter scenario is the case, what in particular distinguishes the writing style of one candidate from another, and how can we infer who the most likely author of a work is from a set of candidates?
Revised: If the latter scenario is the case, what in particular distinguishes the writing style of one candidate from another, and how can we infer who the most likely author of a work is from a set of candidates?

Original: Question 2: Traditional factorization-based embedding methods are rotationally invariant, so that individual dimensions do not have semantic meaning.
Revised: Question 2: Traditional factorization-based embedding methods are rotationally invariant, so that individual dimensions do not have semantic meaning.

Original: How can we break this invariance to yield a model which aligns topics with interpretable dimensions?
Revised: How can we break this invariance to yield a model which aligns topics with interpretable dimensions?

Original: There has been much interest in the differences in language and rhetoric that appeal to different demographics.
Revised: There has been much interest in the differences in language and rhetoric that appeal to different demographics.

Original: For example, studies have been done regarding “ideological signatures” speciﬁc to voters by partisan alignment (Robinson et al.)
Revised: For example, studies have been done regarding “ideological signatures” speciﬁc to voters by partisan alignment (Robinson et al.)

Original: in which linguistic differences were proposed along focal axes, such as the “mind versus the body” in texts with more liberal or conservative ideologies.
Revised: in which linguistic differences were proposed along focal axes, such as the “mind versus the body” in texts with more liberal or conservative ideologies.

Original: How can we systematically infer topical differences such as these between different communities?
Revised: How can we systematically infer topical differences such as these between different communities?

Original: Questions such as these, or more broadly covariate-speciﬁc trends in word usage, motivated this study.
Revised: Questions such as these, or more broadly covariate-speciﬁc trends in word usage, motivated this study.

Original: Concretely, our goal is to provide a general framework through which embeddings of sets of objects with co-occurrence structure, as well as the effects of conditioning on particular covariates, can be learned jointly.
Revised: Concretely, our goal is to provide a general framework through which embeddings of sets of objects with co-occurrence structure, as well as the effects of conditioning on particular covariates, can be learned jointly.

Original: As a byproduct, our model also gives natural meaning to the different dimensions of the embeddings, by breaking the rotational symmetry of previous embedding-learning algorithms, such that the resulting vector representations of words and covariates are “topic-aligned”.
Revised: As a byproduct, our model also gives natural meaning to the different dimensions of the embeddings, by breaking the rotational symmetry of previous embedding-learning algorithms, such that the resulting vector representations of words and covariates are “topic-aligned”.

Original: Previous Work Typically, algorithms for learning embeddings rely on the intuition that some function of the co-occurrence statistics is low rank .
Revised: Previous Work Typically, algorithms for learning embeddings rely on the intuition that some function of the co-occurrence statistics is low rank .

Original: Studies such as GloVe and Word2Vec proposed based on minimizing low-rank approximation-error of nonlinear transforms of the co-occurrence statistics.
Revised: Studies such as GloVe and Word2Vec proposed based on minimizing low-rank approximation-error of nonlinear transforms of the co-occurrence statistics.

Original: let A be the n × n matrix with A ij the co-occurrence between words i and j , where co-occurrence is deﬁned as the (possibly weighted) number of times the words occur together in a window of ﬁxed length.
Revised: let A be the n × n matrix with A ij the co-occurrence between words i and j , where co-occurrence is deﬁned as the (possibly weighted) number of times the words occur together in a window of ﬁxed length.

Original: For example, GloVe aimed to ﬁnd vectors v i ∈ R d and biases b i ∈ R such that the loss
Revised: For example, GloVe aimed to ﬁnd vectors v i ∈ R d and biases b i ∈ R such that the loss

Original: [Equation]
Revised: [Equation]

Original: was minimized, where f was some ﬁxed increasing weight function.
Revised: was minimized, where f was some ﬁxed increasing weight function.

Original: Word2Vec aimed to learn vector representations via minimizing a neural-network based loss function.
Revised: Word2Vec aimed to learn vector representations via minimizing a neural-network based loss function.

Original: A related embedding approach is to directly perform principal component analysis on the PMI (pointwise mutual information) matrix of the words (Bullinaria & Levy).
Revised: A related embedding approach is to directly perform principal component analysis on the PMI (pointwise mutual information) matrix of the words (Bullinaria & Levy).

Original: PMI-factorization based methods aim to ﬁnd vectors { v i } such that
Revised: PMI-factorization based methods aim to ﬁnd vectors { v i } such that

Original: [Equation]
Revised: [Equation]

Original: where the probabilities are taken over the co-occurrence matrix.
Revised: where the probabilities are taken over the co-occurrence matrix.

Original: This is essentially the same as ﬁnding a low-rank matrix V such that V T V ≈ PMI , and empirical results show that the resulting embedding captures useful semantic structure.
Revised: This is essentially the same as ﬁnding a low-rank matrix V such that V T V ≈ PMI , and empirical results show that the resulting embedding captures useful semantic structure.

Original: The ideas of several previous studies on the geometry of word embeddings was helpful in formulating our model.
Revised: The ideas of several previous studies on the geometry of word embeddings was helpful in formulating our model.

Original: A random-walk based mathematical framework for understanding these different successful learning algorithms was proposed (Arora et al., 2016), in which the corpus generation process is a random process driven by the random walk of a discrete-time discourse vector c t ∈ R d .
Revised: A random-walk based mathematical framework for understanding these different successful learning algorithms was proposed (Arora et al., 2016), in which the corpus generation process is a random process driven by the random walk of a discrete-time discourse vector c t ∈ R d .

Original: In this framework, our work can be thought of as analyzing the effects of covariates on the random walk transition kernel and the stationary distribution.
Revised: In this framework, our work can be thought of as analyzing the effects of covariates on the random walk transition kernel and the stationary distribution.

Original: Additionally, there have been previous studies of “multi-sense” word embeddings (Reisinger & Mooney, 2010) (Neelakantan et al., 2015), which is similar to our idea that the same word can have different meanings in different contexts.
Revised: Additionally, there have been previous studies of “multi-sense” word embeddings (Reisinger & Mooney, 2010) (Neelakantan et al., 2015), which is similar to our idea that the same word can have different meanings in different contexts.

Original: However, in the multi-sense setting, the idea is that the word intrinsically has different meanings (for example, “crane” can be an action, a bird, or a vehicle), whereas in ours, the different meanings are imposed by conditioning on a covariate.
Revised: However, in the multi-sense setting, the idea is that the word intrinsically has different meanings (for example, “crane” can be an action, a bird, or a vehicle), whereas in ours, the different meanings are imposed by conditioning on a covariate.

Original: Finally, tensor methods have been used in other settings recently, such as collaborative ﬁltering (Xiong et al., 2010) and (Li & Farias), to learn the effects of conditioning on some summary statistics.
Revised: Finally, tensor methods have been used in other settings recently, such as collaborative ﬁltering (Xiong et al., 2010) and (Li & Farias), to learn the effects of conditioning on some summary statistics.

Original: Our Contributions There are several reasons why a joint learning model based on tensor factorization is more desirable than performing GloVe m times, where m is the number of covariates, so that each covariate-speciﬁc corpus has its own embedding.
Revised: Our Contributions There are several reasons why a joint learning model based on tensor factorization is more desirable than performing GloVe m times, where m is the number of covariates, so that each covariate-speciﬁc corpus has its own embedding.

Original: Our main contributions are a decomposition algorithm that addresses these issues, and the methods for systematic analysis we propose.
Revised: Our main contributions are a decomposition algorithm that addresses these issues, and the methods for systematic analysis we propose.

Original: The ﬁrst issue that arises is sample complexity.
Revised: The ﬁrst issue that arises is sample complexity.

Original: In particular, because for the most part words are used in roughly similar ways across different contexts, the resulting embeddings should not be too different , except perhaps along speciﬁc dimensions.
Revised: In particular, because for the most part words are used in roughly similar ways across different contexts, the resulting embeddings should not be too different , except perhaps along speciﬁc dimensions.

Original: Thus, it is better to jointly train an embedding model along the covariates to aggregate the co-occurrence structure, especially in cases where the entire corpus is large, but many conditional corpuses (conditioned on a covariate) are small.
Revised: Thus, it is better to jointly train an embedding model along the covariates to aggregate the co-occurrence structure, especially in cases where the entire corpus is large, but many conditional corpuses (conditioned on a covariate) are small.

Original: Secondly, simply training a different embedding for each corpus makes it difﬁcult to compare the embeddings across the covariate dimension.
Revised: Secondly, simply training a different embedding for each corpus makes it difﬁcult to compare the embeddings across the covariate dimension.

Original: Because of issues such as rotation invariance of GloVelike models, speciﬁc dimensions mean different things across different runs (and initializations) of these algorithms.
Revised: Because of issues such as rotation invariance of GloVelike models, speciﬁc dimensions mean different things across different runs (and initializations) of these algorithms.

Original: The model we propose has the additional property that it induces a natural basis to view the embeddings in, one which is “topic-aligned” in the sense that it is not rotation-invariant and thus implies independent topic meanings given to different dimensions.
Revised: The model we propose has the additional property that it induces a natural basis to view the embeddings in, one which is “topic-aligned” in the sense that it is not rotation-invariant and thus implies independent topic meanings given to different dimensions.

Original: Paper Organization
Revised: Paper Organization

Original: In section 2, we provide our embedding algorithm, as well as mathematical justiﬁcation for its design.
Revised: In section 2, we provide our embedding algorithm, as well as mathematical justiﬁcation for its design.

Original: In section 3, we detail our dataset.
Revised: In section 3, we detail our dataset.

Original: In section 4, we validate our algorithm with respect to intrinsic properties and standard metrics.
Revised: In section 4, we validate our algorithm with respect to intrinsic properties and standard metrics.

Original: In section 5, we propose several experiments for systematic downstream analysis.
Revised: In section 5, we propose several experiments for systematic downstream analysis.

Original:  T ENSOR C OVARIATE E MBEDDING A LGORITHM
Revised:  T ENSOR C OVARIATE E MBEDDING A LGORITHM

Original: Notation Throughout this section, we will assume a vocabulary of size n and a discrete covariate to condition on of size m (for example, the community that the corpus comes from, i.e. liberal or conservative discussion forums).
Revised: Notation Throughout this section, we will assume a vocabulary of size n and a discrete covariate to condition on of size m (for example, the community that the corpus comes from, i.e. liberal or conservative discussion forums).

Original: It is easy to see how our algorithm generalizes to higher-order tensor decompositions when there are multiple dimensions covariates to condition on (for example, slicing along community and slicing along timeframe simultaneously).
Revised: It is easy to see how our algorithm generalizes to higher-order tensor decompositions when there are multiple dimensions covariates to condition on (for example, slicing along community and slicing along timeframe simultaneously).

Original: Words will be denoted with indices i, j ∈ [ n ] and covariates with index k ∈ [ m ] .
Revised: Words will be denoted with indices i, j ∈ [ n ] and covariates with index k ∈ [ m ] .

Original: Dimensions in our embedding are referred to by index t ∈ [ d ] .
Revised: Dimensions in our embedding are referred to by index t ∈ [ d ] .

Original: We will denote the co-occurrence tensor as A ∈ R n × n × m , where A ijk denotes how many times words i and j occurred together within a window of some ﬁxed length, in the corpus coming from covariate k .
Revised: We will denote the co-occurrence tensor as A ∈ R n × n × m , where A ijk denotes how many times words i and j occurred together within a window of some ﬁxed length, in the corpus coming from covariate k .

Original: The result of our algorithm will be two sets of vectors, { v i ∈ R d } and { c k ∈ R d } , as well as bias terms that also ﬁt into the objective.
Revised: The result of our algorithm will be two sets of vectors, { v i ∈ R d } and { c k ∈ R d } , as well as bias terms that also ﬁt into the objective.

Original: Finally, let (cid:12) denote the element-wise product between two vectors.
Revised: Finally, let (cid:12) denote the element-wise product between two vectors.

Original: Objective Function and Discussion
Revised: Objective Function and Discussion

Original: Here, we give the objective function our method minimizes, and provide some explanation for how one should imagine the effect of the covariate weights.
Revised: Here, we give the objective function our method minimizes, and provide some explanation for how one should imagine the effect of the covariate weights.

Original: The objective function we minimize is the following partial non-negative tensor factorization objective function for jointly training word vectors and weight vectors representing the effect of covariates, adapted from the original GloVe objective (note that
Revised: The objective function we minimize is the following partial non-negative tensor factorization objective function for jointly training word vectors and weight vectors representing the effect of covariates, adapted from the original GloVe objective (note that

Original: c k (cid:12) v i = diag ( c k ) v i , where diag ( c k ) is the diagonal matrix weighting of covariate k ):
Revised: c k (cid:12) v i = diag ( c k ) v i , where diag ( c k ) is the diagonal matrix weighting of covariate k ):

Original: [Equation]
Revised: [Equation]

Original: which is to be optimized over { v i ∈ R d } , { c k ∈ R d } , and { b ik ∈ R } .
Revised: which is to be optimized over { v i ∈ R d } , { c k ∈ R d } , and { b ik ∈ R } .

Original: To gain a little more intuition for why this is a reasonable objective function, note that the resulting objective for a single “covariate slice” is essentially
Revised: To gain a little more intuition for why this is a reasonable objective function, note that the resulting objective for a single “covariate slice” is essentially

Original: [Equation]
Revised: [Equation]

Original: which ﬁts the vectors c k (cid:12) v i to the data, thus approximating the statistic log A ijk with (cid:80) model since the c k can be absorbed into the v i
Revised: which ﬁts the vectors c k (cid:12) v i to the data, thus approximating the statistic log A ijk with (cid:80) model since the c k can be absorbed into the v i

Original: We used f ( x ) =
Revised: We used f ( x ) =

Original: ( min(100 ,x )) 0 .
Revised: ( min(100 ,x )) 0 .

Original: 75 , to parallel the original objective function in (Pennington et al., 2014).
Revised: 75 , to parallel the original objective function in (Pennington et al., 2014).

Original: One can think of the dimensions our model learns as independent topics, and the effects of the covariate weights c k as upweighting or downweighting the importance of these topics in contributing to the conditional co-occurrence statistics.
Revised: One can think of the dimensions our model learns as independent topics, and the effects of the covariate weights c k as upweighting or downweighting the importance of these topics in contributing to the conditional co-occurrence statistics.

Original: A Geometric View of Embeddings and Tensor Decomposition
Revised: A Geometric View of Embeddings and Tensor Decomposition

Original: Here we provide a geometric perspective on our model in the context of some prior work.
Revised: Here we provide a geometric perspective on our model in the context of some prior work.

Original: This geometric interpretation is not necessary to applying our method and to understand its results.
Revised: This geometric interpretation is not necessary to applying our method and to understand its results.

Original: Throughout this section, note that at a high level, the aim of our method is to learn sets { v i ∈ R d } , { c k ∈ R d } , and { b ik ∈ R } for 1 ≤ i ≤ n, 1 ≤ k ≤ m , such that for a ﬁxed k , the vectors { c k (cid:12) v i } and the biases { b ik } approximate the vectors and biases that would have been learned from running GloVe on only the k th slice of the co-occurrence tensor.
Revised: Throughout this section, note that at a high level, the aim of our method is to learn sets { v i ∈ R d } , { c k ∈ R d } , and { b ik ∈ R } for 1 ≤ i ≤ n, 1 ≤ k ≤ m , such that for a ﬁxed k , the vectors { c k (cid:12) v i } and the biases { b ik } approximate the vectors and biases that would have been learned from running GloVe on only the k th slice of the co-occurrence tensor.

Original: We now provide some rationale for why this is a reasonable objective.
Revised: We now provide some rationale for why this is a reasonable objective.

Original: The main motivation for our algorithm is the following geometric intuition inspired in part by the model of Arora et al.
Revised: The main motivation for our algorithm is the following geometric intuition inspired in part by the model of Arora et al.

Original: In their model, corpus generation is determined by the nature of the random walk performed by a context vector c t over time steps.
Revised: In their model, corpus generation is determined by the nature of the random walk performed by a context vector c t over time steps.

Original: In particular, for each timestep t , the context vector c t emits words i with probability ∝ exp( v i · c t ) , and then c t updates its location based on a transition distribution.
Revised: In particular, for each timestep t , the context vector c t emits words i with probability ∝ exp( v i · c t ) , and then c t updates its location based on a transition distribution.

Original: The assumption is that the transition distribution has a stationary distribution that is uniform over some ellipse.
Revised: The assumption is that the transition distribution has a stationary distribution that is uniform over some ellipse.

Original: One natural equivalence family of transition matrices that preserves this property is the group resulting from multiplication by a positive semideﬁnite matrix.
Revised: One natural equivalence family of transition matrices that preserves this property is the group resulting from multiplication by a positive semideﬁnite matrix.

Original: Therefore, we consider a natural extension of this model, where the embedding resulting from conditioning on different covariates is equivalent to multiplying the transition matrix by a symmetric
Revised: Therefore, we consider a natural extension of this model, where the embedding resulting from conditioning on different covariates is equivalent to multiplying the transition matrix by a symmetric

Original: PSD matrix.
Revised: PSD matrix.

Original: Alternatively, this is equivalent to a model where the transition kernel remains unchanged, but the embedding vectors themselves are multiplied by a symmetric PSD matrix (namely, the Moore-Penrose pseudoinverse of the original PSD matrix).
Revised: Alternatively, this is equivalent to a model where the transition kernel remains unchanged, but the embedding vectors themselves are multiplied by a symmetric PSD matrix (namely, the Moore-Penrose pseudoinverse of the original PSD matrix).

Original: This is the viewpoint we adopt.
Revised: This is the viewpoint we adopt.

Original: In this framework, assign each covariate k its own symmetric PSD matrix, B k .
Revised: In this framework, assign each covariate k its own symmetric PSD matrix, B k .

Original: It is well-known that any symmetric PSD matrix can be factorized as B k
Revised: It is well-known that any symmetric PSD matrix can be factorized as B k

Original: = R T k D k R k for some orthonormal basis R k and some (nonnegative) diagonal D k .
Revised: = R T k D k R k for some orthonormal basis R k and some (nonnegative) diagonal D k .

Original: Thus it sufﬁces to consider the effect of a covariate on some ground truth “base embedding” M as applying the linear operator B k to each embedding vector,
Revised: Thus it sufﬁces to consider the effect of a covariate on some ground truth “base embedding” M as applying the linear operator B k to each embedding vector,

Original: This model is quite expressive in its own right, but we consider a natural restriction where we propose that there exists some universal (at least across covariates) basis R under which the resulting embeddings are affected by covariates via multiplication by just a diagonal matrix, instead of a PSD matrix.
Revised: This model is quite expressive in its own right, but we consider a natural restriction where we propose that there exists some universal (at least across covariates) basis R under which the resulting embeddings are affected by covariates via multiplication by just a diagonal matrix, instead of a PSD matrix.

Original: In particular, we note that
Revised: In particular, we note that

Original: [Equation]
Revised: [Equation]

Original: where M (cid:48) k
Revised: where M (cid:48) k

Original: = R k M is a rotated version of M .
Revised: = R k M is a rotated version of M .

Original: Now, in the restricted model where all the R k are equal, we can write all the M (cid:48) k as RM , so it sufﬁces to just consider the rotation of the basis that the original embedding was trained in where R is just the identity (since matrix-factorization based word embedding models are rotation invariant).
Revised: Now, in the restricted model where all the R k are equal, we can write all the M (cid:48) k as RM , so it sufﬁces to just consider the rotation of the basis that the original embedding was trained in where R is just the identity (since matrix-factorization based word embedding models are rotation invariant).

Original: Under this model, the co-occurrence statistics under some transformation should be equivalent to M T D 2 k M .
Revised: Under this model, the co-occurrence statistics under some transformation should be equivalent to M T D 2 k M .

Original: A careful comparison shows that this approximation is precisely that which is implied by equation 4, as desired.
Revised: A careful comparison shows that this approximation is precisely that which is implied by equation 4, as desired.

Original: Note that this is essentially saying that in this distributed word representation model, there exists some rotation of the embedding space under which the effect of the covariate separates along dimensions.
Revised: Note that this is essentially saying that in this distributed word representation model, there exists some rotation of the embedding space under which the effect of the covariate separates along dimensions.

Original: The implication is that there are some set of independent “topics” that each covariate will upweight or downweight in importance (or possibly ignore altogether with a weight of 0), characterizing the effect of this conditioning directly in terms of the effect on these topics.
Revised: The implication is that there are some set of independent “topics” that each covariate will upweight or downweight in importance (or possibly ignore altogether with a weight of 0), characterizing the effect of this conditioning directly in terms of the effect on these topics.

Original: Algorithm Details
Revised: Algorithm Details

Original: Our model learns the resulting parameters { v i ∈ R d } , { c k ∈ R d } , and { b ik } , by using the Adam (Kingma & Ba, 2014) algorithm, which was empirically shown to yield good convergence results in the original GloVe setting.
Revised: Our model learns the resulting parameters { v i ∈ R d } , { c k ∈ R d } , and { b ik } , by using the Adam (Kingma & Ba, 2014) algorithm, which was empirically shown to yield good convergence results in the original GloVe setting.

Original: The speciﬁc hyperparameters used for each dataset will be described in the next section.
Revised: The speciﬁc hyperparameters used for each dataset will be described in the next section.

Original: The word and covariate weight vectors were initialized as random unit vectors 1 .
Revised: The word and covariate weight vectors were initialized as random unit vectors 1 .

Original:  D ATASET
Revised:  D ATASET

Original: We evaluated our method in to primary datasets.
Revised: We evaluated our method in to primary datasets.

Original: In both datasets, co-occurrence statistics were formed by considering size 8 windows and using an inverse-distance weighting (e.g. neighboring words had 1 added to their co-occurrence, and words 3 apart had 13 added), which was suggested by some implementations of (Pennington et al., 2014).
Revised: In both datasets, co-occurrence statistics were formed by considering size 8 windows and using an inverse-distance weighting (e.g. neighboring words had 1 added to their co-occurrence, and words 3 apart had 13 added), which was suggested by some implementations of (Pennington et al., 2014).

Original: The ﬁrst dataset, referred to as the “book dataset”, consists of the full text from 29 books written by 4 different authors.
Revised: The ﬁrst dataset, referred to as the “book dataset”, consists of the full text from 29 books written by 4 different authors.

Original: The books we used were J.K. Rowling’s
Revised: The books we used were J.K. Rowling’s

Original: “Harry Potter” series (7 books), “Cormoran Strike” series (3 books), and “The Casual Vacancy”; C. S. Lewis’s “The Chronicles of
Revised: “Harry Potter” series (7 books), “Cormoran Strike” series (3 books), and “The Casual Vacancy”; C. S. Lewis’s “The Chronicles of

Original: Narnia” series (7 books), and “The Screwtape Letters”; George R. R. Martin’s “A Song of Ice and
Revised: Narnia” series (7 books), and “The Screwtape Letters”; George R. R. Martin’s “A Song of Ice and

Original: Fire” series (5 books); and Stephenie Meyer’s “Twilight” series (4 books), and “The Host”.
Revised: Fire” series (5 books); and Stephenie Meyer’s “Twilight” series (4 books), and “The Host”.

Original: These books are ﬁction works in similar genres, with highly overlapping vocabularies and common themes.
Revised: These books are ﬁction works in similar genres, with highly overlapping vocabularies and common themes.

Original: A trivial way of learning series-speciﬁc tendencies in word usage would be to cluster according to unique vocabularies (for example, only the “Harry Potter” series would have words such as “Harry” and “Ron” frequently), so the co-occurrence tensor was formed by looking at all words that occurred in all of the series with multiple books, which eliminated all series-speciﬁc words.
Revised: A trivial way of learning series-speciﬁc tendencies in word usage would be to cluster according to unique vocabularies (for example, only the “Harry Potter” series would have words such as “Harry” and “Ron” frequently), so the co-occurrence tensor was formed by looking at all words that occurred in all of the series with multiple books, which eliminated all series-speciﬁc words.

Original: Furthermore, series by the same author had very different themes, so there is no reason intrinsic to the vocabulary to believe the weight vectors would cluster by author.
Revised: Furthermore, series by the same author had very different themes, so there is no reason intrinsic to the vocabulary to believe the weight vectors would cluster by author.

Original: The vocabulary size was 5,020, and after tuning our algorithm to embed this dataset, we used 100 dimensions and a learning rate of 10 − 5 .
Revised: The vocabulary size was 5,020, and after tuning our algorithm to embed this dataset, we used 100 dimensions and a learning rate of 10 − 5 .

Original: The second dataset, referred to as the “politics dataset”, was a collection of comments made in 2016 in 6 different subreddits on the popular discussion forum reddit, and was selected to address both Questions 1 and 2.
Revised: The second dataset, referred to as the “politics dataset”, was a collection of comments made in 2016 in 6 different subreddits on the popular discussion forum reddit, and was selected to address both Questions 1 and 2.

Original: The covariate was the discussion forum, and the subreddits we used were
Revised: The covariate was the discussion forum, and the subreddits we used were

Original: AskReddit , news , politics , SandersForPresident , The Donald , and WorldNews .
Revised: AskReddit , news , politics , SandersForPresident , The Donald , and WorldNews .

Original: AskReddit was chosen as a baseline discussion forum with a very general vocabulary usage, and the discussion forums for the Sanders and Trump support bases were also selected, as well as three politically-relevant but more neutral communities (it should be noted that the politics discussion forum tends to be very leftleaning).
Revised: AskReddit was chosen as a baseline discussion forum with a very general vocabulary usage, and the discussion forums for the Sanders and Trump support bases were also selected, as well as three politically-relevant but more neutral communities (it should be noted that the politics discussion forum tends to be very leftleaning).

Original: We considered a vocabulary of size 15,000, after removing the 28 most common words (suggested by other works in the embedding literature) and entries of the cooccurrence tensor with less than 10 occurrences (for the sake of training efﬁciency).
Revised: We considered a vocabulary of size 15,000, after removing the 28 most common words (suggested by other works in the embedding literature) and entries of the cooccurrence tensor with less than 10 occurrences (for the sake of training efﬁciency).

Original: The embedding used 200 dimensions and a learning rate of 10 − 5 .
Revised: The embedding used 200 dimensions and a learning rate of 10 − 5 .

Original:  E XPERIMENTAL V ALIDATION 4. C LUSTERING BY W EIGHTS
Revised:  E XPERIMENTAL V ALIDATION 4. C LUSTERING BY W EIGHTS

Original: We performed the tensor decomposition algorithm on the book dataset, and considered how well the weight vectors of the covariate clustered by series and also by author.
Revised: We performed the tensor decomposition algorithm on the book dataset, and considered how well the weight vectors of the covariate clustered by series and also by author.

Original: [Figure]
Revised: [Figure]

Original: For every book in every series, the closest weight vectors by series were all the books in the same series.
Revised: For every book in every series, the closest weight vectors by series were all the books in the same series.

Original: Furthermore, for every book written by every author, the closest weight vectors by author were all the books by the same author.
Revised: Furthermore, for every book written by every author, the closest weight vectors by author were all the books by the same author.

Original: This clear clustering behavior even when only conditioning on co-occurrence statistics for words that appear in all series (throwing out series-speciﬁc terms) implies the ability of the weight vectors to cluster according to higher-order information, perhaps such as writing style.
Revised: This clear clustering behavior even when only conditioning on co-occurrence statistics for words that appear in all series (throwing out series-speciﬁc terms) implies the ability of the weight vectors to cluster according to higher-order information, perhaps such as writing style.

Original: As a baseline, we considered the topic breakdown on the same co-occurrence statistics predicted by Latent Dirichlet Allocation (Blei et al., 2003) with various dimensionalities (20, 50, 100), all of which failed to produce any meaningful clusters.
Revised: As a baseline, we considered the topic breakdown on the same co-occurrence statistics predicted by Latent Dirichlet Allocation (Blei et al., 2003) with various dimensionalities (20, 50, 100), all of which failed to produce any meaningful clusters.

Original: 4. D ATA E FFICIENCY AND V ALIDATION
Revised: 4. D ATA E FFICIENCY AND V ALIDATION

Original: Consider the problem of learning an embedding for the text of a particular book (or series).
Revised: Consider the problem of learning an embedding for the text of a particular book (or series).

Original: The main advantage given by using the covariate-speciﬁc embedding that our method yields over applying
Revised: The main advantage given by using the covariate-speciﬁc embedding that our method yields over applying

Original: GloVe to the individual slice is a data efﬁciency - by pooling the co-occurrence statistics of words across other books, we are able to give better (less noisy) estimates of the vectors, especially for sparsely-occurring or nonexistent words.
Revised: GloVe to the individual slice is a data efﬁciency - by pooling the co-occurrence statistics of words across other books, we are able to give better (less noisy) estimates of the vectors, especially for sparsely-occurring or nonexistent words.

Original: For reference, individual books contained betweenand 355814 words.
Revised: For reference, individual books contained betweenand 355814 words.

Original: To this end, we performed the following experiment.
Revised: To this end, we performed the following experiment.

Original: For some book k , consider two embeddings: 1) the result of performing GloVe on the co-occurrence statistics of just the book, and 2) the (weighted) embedding resulting from our algorithm, speciﬁc to the book.
Revised: For some book k , consider two embeddings: 1) the result of performing GloVe on the co-occurrence statistics of just the book, and 2) the (weighted) embedding resulting from our algorithm, speciﬁc to the book.

Original: Then, we tested these resulting embeddings using a standard suite of evaluation metrics, including cluster purity and correlation similarities.
Revised: Then, we tested these resulting embeddings using a standard suite of evaluation metrics, including cluster purity and correlation similarities.

Original: Our method outperformed method 1 on all 7 tasks, often by a signiﬁcant margin.
Revised: Our method outperformed method 1 on all 7 tasks, often by a signiﬁcant margin.

Original: [Table]
Revised: [Table]

Original: 4. S PARSITY OF W EIGHT V ECTORS
Revised: 4. S PARSITY OF W EIGHT V ECTORS

Original: Because of experimentally veriﬁed isotropic distributional properties of word embedding vectors (Arora et al., 2016), it is unreasonable to ask for sparsity in the word embeddings themselves.
Revised: Because of experimentally veriﬁed isotropic distributional properties of word embedding vectors (Arora et al., 2016), it is unreasonable to ask for sparsity in the word embeddings themselves.

Original: However, our topic-speciﬁc weighting scheme for the covariate weight vectors implies that sparsity is desirable in the weights.
Revised: However, our topic-speciﬁc weighting scheme for the covariate weight vectors implies that sparsity is desirable in the weights.

Original: The weight sparsity resulting from our algorithm was experimentally veriﬁed through many runs of our embedding method, as well as across different optimization methods.
Revised: The weight sparsity resulting from our algorithm was experimentally veriﬁed through many runs of our embedding method, as well as across different optimization methods.

Original: Note that in the objective function (3), sparse coordinates will become “stuck” at 0, because the gradient update of c kt is proportional to c kt :
Revised: Note that in the objective function (3), sparse coordinates will become “stuck” at 0, because the gradient update of c kt is proportional to c kt :

Original: [Equation]
Revised: [Equation]

Original: The dimensions that the covariates were sparse in did not overlap by much: the average number of sparse coordinates per weight vector was 20.7, and the average number of coordinates that was sparse in both vectors of a random pair was 5.2.
Revised: The dimensions that the covariates were sparse in did not overlap by much: the average number of sparse coordinates per weight vector was 20.7, and the average number of coordinates that was sparse in both vectors of a random pair was 5.2.

Original: This suggest that the set of “topics” each conditional slice of the tensor does not care about is fairly speciﬁc.
Revised: This suggest that the set of “topics” each conditional slice of the tensor does not care about is fairly speciﬁc.

Original: Experimenting with regularizing the word vectors forced the model into a smaller subspace (i.e. sparse dimensions existed but were shared across all the words), which is not useful and provides further evidence for the natural isotropic distribution of word vectors.
Revised: Experimenting with regularizing the word vectors forced the model into a smaller subspace (i.e. sparse dimensions existed but were shared across all the words), which is not useful and provides further evidence for the natural isotropic distribution of word vectors.

Original: Regularizing the weights of the covariate vectors by adding an lpenalty term did not noticeably change the sparsity pattern.
Revised: Regularizing the weights of the covariate vectors by adding an lpenalty term did not noticeably change the sparsity pattern.

Original: [Table]
Revised: [Table]

Original: Donald worldnews
Revised: Donald worldnews

Original: [Table]
Revised: [Table]

Original: To conﬁrm that the sparsity was a result of separation of covariate effects rather than an artifact of our algorithm, we ran our decomposition on a co-occurrence tensor which was the result of taking the same slice (subreddit) and subsampling its entries 3 times, creating 3 slices of essentially similar cooccurrence statistics.
Revised: To conﬁrm that the sparsity was a result of separation of covariate effects rather than an artifact of our algorithm, we ran our decomposition on a co-occurrence tensor which was the result of taking the same slice (subreddit) and subsampling its entries 3 times, creating 3 slices of essentially similar cooccurrence statistics.

Original: We applied our algorithm with different learning parameters, and the resulting weight vectors after 90 iterations (when the outputs vectors converged) were extremely non-sparse, with between 0 and 2 sparse coordinates per weight vector.
Revised: We applied our algorithm with different learning parameters, and the resulting weight vectors after 90 iterations (when the outputs vectors converged) were extremely non-sparse, with between 0 and 2 sparse coordinates per weight vector.

Original: The dimensions that are speciﬁcallyfor a covariate corresponds to topics that are relatively less relevant for that covariate.
Revised: The dimensions that are speciﬁcallyfor a covariate corresponds to topics that are relatively less relevant for that covariate.

Original: In the next section, we develop methods to systematically interpret the covariate weights in terms of topics.
Revised: In the next section, we develop methods to systematically interpret the covariate weights in terms of topics.

Original:  I NTERPRETATION 5. I NFERENCE OF T OPIC M EANING
Revised:  I NTERPRETATION 5. I NFERENCE OF T OPIC M EANING

Original: A simple test of inferring topic meaning (i.e. topics coordinates are associated with) is to consider the set of words which are large in the given coordinate.
Revised: A simple test of inferring topic meaning (i.e. topics coordinates are associated with) is to consider the set of words which are large in the given coordinate.

Original: Concretely the task is, given some index t ∈
Revised: Concretely the task is, given some index t ∈

Original: [ d ] , output the words whose (normalized) vectors have largest value in dimension t .
Revised: [ d ] , output the words whose (normalized) vectors have largest value in dimension t .

Original: We show the results of this experiment for several of the sparse coordinates in the AskReddit weight vector:
Revised: We show the results of this experiment for several of the sparse coordinates in the AskReddit weight vector:

Original: [Table]
Revised: [Table]

Original: There are several conclusions to be drawn from this experiment.
Revised: There are several conclusions to be drawn from this experiment.

Original: Firstly, while there is some clear noise, speciﬁc topic meanings do seem to appear in certain coordinates (which we infer in the table above).
Revised: Firstly, while there is some clear noise, speciﬁc topic meanings do seem to appear in certain coordinates (which we infer in the table above).

Original: It is reasonable that meaning would appear out of coordinates which are sparsely weighted in some covariate, because this means that it is a topic that is relevant in other discussion forums but purposely ignored in some speciﬁc forum, so it is likely to have a consistent theme.
Revised: It is reasonable that meaning would appear out of coordinates which are sparsely weighted in some covariate, because this means that it is a topic that is relevant in other discussion forums but purposely ignored in some speciﬁc forum, so it is likely to have a consistent theme.

Original: When we performed this task for coordinates which had low variance in their weights between covariates, the resulting words were much less consistent.
Revised: When we performed this task for coordinates which had low variance in their weights between covariates, the resulting words were much less consistent.

Original: It is also interesting to see how covariates weight a topic whose meaning we have identiﬁed.
Revised: It is also interesting to see how covariates weight a topic whose meaning we have identiﬁed.

Original: For example, for coordinate 194 (corresponding to military themes), AskReddit placed negligible weight, news and worldnews placed weight 2.06 and 2.04 respectively, SandersForPresident and
Revised: For example, for coordinate 194 (corresponding to military themes), AskReddit placed negligible weight, news and worldnews placed weight 2.06 and 2.04 respectively, SandersForPresident and

Original: The Donald placed weight 0.41 and 0.38 respectively, and politics placed weight 0.05.
Revised: The Donald placed weight 0.41 and 0.38 respectively, and politics placed weight 0.05.

Original: This process can also be reversed - for example, consider coordinates small in worldnews and large in news .
Revised: This process can also be reversed - for example, consider coordinates small in worldnews and large in news .

Original: One example was coordinate 188, and upon checking the words that were large in this coordinate ( { taser, troops, supremacists, underpaid, rioters, amendment, racially, hispanic } )
Revised: One example was coordinate 188, and upon checking the words that were large in this coordinate ( { taser, troops, supremacists, underpaid, rioters, amendment, racially, hispanic } )

Original: it seemed clear that it had themes of rights and protests, which makes sense as a domestic issue, not a global one.
Revised: it seemed clear that it had themes of rights and protests, which makes sense as a domestic issue, not a global one.

Original: 5. T OPICAL W ORD D RIFT
Revised: 5. T OPICAL W ORD D RIFT

Original: We performed the following experiment: which pairs of words start off close in the baseline embedding, yet under some covariate weights move far apart (or vice versa)?
Revised: We performed the following experiment: which pairs of words start off close in the baseline embedding, yet under some covariate weights move far apart (or vice versa)?

Original: Concretely, the task is, for a ﬁxed word
Revised: Concretely, the task is, for a ﬁxed word

Original: i and a ﬁxed covariate k , identify words j such that || c k (cid:12) v
Revised: i and a ﬁxed covariate k , identify words j such that || c k (cid:12) v

Original: i − c k (cid:12) v j || (cid:29) ||
Revised: i − c k (cid:12) v j || (cid:29) ||

Original: v i − v j ||
Revised: v i − v j ||

Original: or || c k (cid:12) v
Revised: or || c k (cid:12) v

Original: i − c k (cid:12) v j || (cid:28) ||
Revised: i − c k (cid:12) v j || (cid:28) ||

Original: v i − v j || , where the magnitude of drift is quantiﬁed by the ratio of distances in the normalized embedding.
Revised: v i − v j || , where the magnitude of drift is quantiﬁed by the ratio of distances in the normalized embedding.

Original: The motivation is to ﬁnd words whose general usage is similar, but have very different meanings in speciﬁc communities.
Revised: The motivation is to ﬁnd words whose general usage is similar, but have very different meanings in speciﬁc communities.

Original: We present a few representative examples of this experiment below, for k =
Revised: We present a few representative examples of this experiment below, for k =

Original: The Donald .
Revised: The Donald .

Original: [Table]
Revised: [Table]

Original: affecting
Revised: affecting

Original: [Table]
Revised: [Table]

Original: policy
Revised: policy

Original: [Table]
Revised: [Table]

Original: attorneys
Revised: attorneys

Original: [Table]
Revised: [Table]

Original: protect
Revised: protect

Original: Combining the previous two sections allows us to do an end-to-end case study on words that drift under a covariate, so we can explain speciﬁcally which topics (under reweighting) caused this shift.
Revised: Combining the previous two sections allows us to do an end-to-end case study on words that drift under a covariate, so we can explain speciﬁcally which topics (under reweighting) caused this shift.

Original: For example, the words “immigrant” and “parasite” were signiﬁcantly closer under the weights placed by The Donald , so we considered dimensions that were simultaneously large in the vector v immigrant − v parasite and sparse in the weight vector c
Revised: For example, the words “immigrant” and “parasite” were signiﬁcantly closer under the weights placed by The Donald , so we considered dimensions that were simultaneously large in the vector v immigrant − v parasite and sparse in the weight vector c

Original: The Donald .
Revised: The Donald .

Original: The dimensions 89 andwere sparse and also the 2nd and 3rd largest coordinates in the difference vector, so they had a large contribution to the subspace which was zeroed out under the reweighting.
Revised: The dimensions 89 andwere sparse and also the 2nd and 3rd largest coordinates in the difference vector, so they had a large contribution to the subspace which was zeroed out under the reweighting.

Original: Words that were large in these dimensions (and thus representative of the zeroed out subspace meaning) included { misunderstanding, populace, scapegoat, rebuilding } for 89, and { answers, barriers, applicants, backstory, selﬂess, indigenous } for 139.
Revised: Words that were large in these dimensions (and thus representative of the zeroed out subspace meaning) included { misunderstanding, populace, scapegoat, rebuilding } for 89, and { answers, barriers, applicants, backstory, selﬂess, indigenous } for 139.

Original: This suggests two independent reasons for the drift, namely dimensions corresponding to emotional appeal and legal immigration respectively being zeroed out.
Revised: This suggests two independent reasons for the drift, namely dimensions corresponding to emotional appeal and legal immigration respectively being zeroed out.

Original: 5. C OVARIATE -S PECIFIC A NALOGIES
Revised: 5. C OVARIATE -S PECIFIC A NALOGIES

Original: One of the most famous downstream applications of recent embedding methods such as (Pennington et al., 2014) and (Mikolov et al., 2013) is their ability to capture analogies.
Revised: One of the most famous downstream applications of recent embedding methods such as (Pennington et al., 2014) and (Mikolov et al., 2013) is their ability to capture analogies.

Original: This is typically formulated as a : b :: c : d ↔ v a − v b
Revised: This is typically formulated as a : b :: c : d ↔ v a − v b

Original: ≈ v c − v d , for example v woman − v queen ≈ v man − v king .
Revised: ≈ v c − v d , for example v woman − v queen ≈ v man − v king .

Original: We considered how well our method captured covariate-speciﬁc analogies , which appear in a covariatespeciﬁc embedding but not most others.
Revised: We considered how well our method captured covariate-speciﬁc analogies , which appear in a covariatespeciﬁc embedding but not most others.

Original: To this end, we considered experiments of the form: for ﬁxed words a, b, c , determine words d such that for some covariate k , the quantity
Revised: To this end, we considered experiments of the form: for ﬁxed words a, b, c , determine words d such that for some covariate k , the quantity

Original: [Equation]
Revised: [Equation]

Original: is small, yet for other k , the quantity is large.
Revised: is small, yet for other k , the quantity is large.

Original: The intuition is that under the covariate transform, v c − v d points roughly in the same direction as v a − v b , and d is close to c in semantic meaning.
Revised: The intuition is that under the covariate transform, v c − v d points roughly in the same direction as v a − v b , and d is close to c in semantic meaning.

Original: In particular, we set a = “hillary”, c = “trump”, and found words b for which there existed a d consistently at the top across subreddits (implying existence of strong analogies).
Revised: In particular, we set a = “hillary”, c = “trump”, and found words b for which there existed a d consistently at the top across subreddits (implying existence of strong analogies).

Original: For example, when b = “woman”, d = “man” was the best analogy for every weighting.
Revised: For example, when b = “woman”, d = “man” was the best analogy for every weighting.

Original: Then, for these b , we considered words d whose relative rankings in the subreddits had high variance.
Revised: Then, for these b , we considered words d whose relative rankings in the subreddits had high variance.

Original: The differential analogies captured were quite striking, and we present several representative examples in Table 5.
Revised: The differential analogies captured were quite striking, and we present several representative examples in Table 5.

Original: [Table]
Revised: [Table]

Original: Discussion We have presented a joint tensor model that essentially learns an embedding for each word and for each covariate.
Revised: Discussion We have presented a joint tensor model that essentially learns an embedding for each word and for each covariate.

Original: This model makes it very simple to compute the covariate speciﬁc embedding: we just take the element-wise vector product.
Revised: This model makes it very simple to compute the covariate speciﬁc embedding: we just take the element-wise vector product.

Original: It also enables us to systematically interpret the covariate vector by looking at dimensions along which weight is large or 0.
Revised: It also enables us to systematically interpret the covariate vector by looking at dimensions along which weight is large or 0.

Original: Our experiments show that these dimensions can be interpreted as coherent topics.
Revised: Our experiments show that these dimensions can be interpreted as coherent topics.

Original: While we focus on word embeddings, our tensor covariate embedding model can be naturally applied in other settings.
Revised: While we focus on word embeddings, our tensor covariate embedding model can be naturally applied in other settings.

Original: For example, there is a large amount of interest in learning embeddings of individual genes to capture biological interactions.
Revised: For example, there is a large amount of interest in learning embeddings of individual genes to capture biological interactions.

Original: The natural covariates here are the cell types and our method would be able to model cell-type speciﬁc gene interactions.
Revised: The natural covariates here are the cell types and our method would be able to model cell-type speciﬁc gene interactions.

Original: Another interesting setting with conditional covariates would be time-series speciﬁc embeddings, where data efﬁciency becomes more of an issue.
Revised: Another interesting setting with conditional covariates would be time-series speciﬁc embeddings, where data efﬁciency becomes more of an issue.

Original: We hope our framework is general enough that it will be of use to practitioners in these settings and others.
Revised: We hope our framework is general enough that it will be of use to practitioners in these settings and others.

Original: B A LGORITHM SETTING NOTES
Revised: B A LGORITHM SETTING NOTES

Original: [Figure]
Revised: [Figure]

Original: We also experimented with using (Duchi et al.)
Revised: We also experimented with using (Duchi et al.)

Original: as the optimization method, but the resulting weight vectors in the politics dataset had highly-overlapping sparse dimensions.
Revised: as the optimization method, but the resulting weight vectors in the politics dataset had highly-overlapping sparse dimensions.

Original: This implies that the optimization method tried to ﬁt the model to a smaller-dimensional subspace, which is not a desirable source of sparsity.
Revised: This implies that the optimization method tried to ﬁt the model to a smaller-dimensional subspace, which is not a desirable source of sparsity.