M ODEL C OMPARISON FOR S EMANTIC G ROUPING
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a probabilistic framework for quantifying the semantic similarity between two groups of embeddings.
We formulate the task of semantic similarity as a model comparison task in which we contrast a generative model which jointly models two sentences versus one that does not.
We illustrate how this framework can be used for the Semantic Textual Similarity tasks using clear assumptions about how the embeddings of words are generated.
We apply information criteria based model comparison to overcome the shortcomings of Bayesian model comparison, whilst still penalising model complexity.
We achieve competitive results by applying the proposed framework with an appropriate choice of likelihood on the STS datasets.
I NTRODUCTION
The problem of Semantic Textual Similarity (STS), measuring how closely the meaning of one piece of text corresponds to that of another, has been studied in the hope of improving performance across various problems in Natural Language Processing (NLP), including information retrieval (Zheng & Callan, 2015).
Recent progress in the generation of word embeddings (Mikolov et al., 2013) has allowed the encoding of words using distributed vector representations, which capture semantic information through their location in the learned embedding space.
Despite the extent to which semantic relations between words are captured in this space, it remains a challenge to adapt these individual word embeddings to express semantic similarity between word groups, like documents, sentences, and other textual formats.
Recent methods for STS rely on additive composition of word vectors (Arora et al., 2016; Blacoe & Lapata, 2012; Mitchell & Lapata, 2008; 2010) or deep learning architectures (Kiros et al., 2015), which summarise a sentence through a single embedding.
The resulting sentence vectors are then compared using cosine similarity — a choice that is not theoretically justiﬁed, stemming only from the fact that cosine similarity gives good empirical results.
It is difﬁcult for a practitioner to utilize word vectors efﬁciently if the underlying assumptions in the similarity measure are not well understood.
The main contribution of our work is the proposal of a framework that addresses these issues by explicitly deriving the similarity measure through a chosen generative model of embeddings, instead of empirically motivating it. Via this design process, a practitioner can encode suitable assumptions and constraints that may be favourable to the application of interest. Furthermore, this framework puts forward a new research direction that could help understand semantic similarity, in which practitioners can study suitable embedding distributions and assess how these perform. The secondary contribution of our work is a derivation of a similarity measure that performs well in an online setting 1 .
Online settings are both practical and key to use-cases that involve information retrieval in dialogue systems.
For example, in a chat-bot application new queries will arrive one at a time and methods such as the one proposed in Arora et al. (2016) will not perform as strongly as they do on the benchmark datasets. This is because one cannot perform the required data preprocessing on the entire query dataset, which will not be available a priori in online settings. Whilst our framework produces an online similarity metric, it remains competitive to ofﬂine methods such as Arora et al.
We achieve results comparable to Arora et al. (2016) on the STS dataset in O ( nd ) time — compared to the O ( nd 2 ) average complexity of their method (where n is the number of words in a sentence, and d is the embedding size).  B ACKGROUND
The compositional nature of distributed representations demonstrated in Mikolov et al.
and Pennington et al.
(2014) indicate the presence of semantic groups in the representation space of word embeddings; an idea which has been further explored in Athiwaratkun & Wilson (2017).
Under this assumption, the task of semantic similarity can be formulated as the following question: “Are the two sentences (as groups of words) partitions of the same semantic group?”
Figure 1 illustrates this concept.
Utilising this rephrasing, this work formulates the task of semantic similarity between two arbitrary groups of objects as a model comparison problem.
Taking inspiration from Ghahramani & Heller (2006) and Marshall et al.
we propose the generative models for groups (e.g. sentences) D 1 , D 2 seen in Figure 2.
The Bayes Factor for this graphical model is then formally deﬁned as
[Equation]
To obtain the evidence p ( D|M i ) the parameters of the model must be marginalised
[Equation]
where ⊕ denotes concatenation.
Computing the semantic similarity score of the two groups D 1 , D 2 under the Bayesian framework requires selecting a reasonable model likelihood p ( w k | θ ) , prior density on the parameters p ( θ ) , and computing the marginal evidence speciﬁed above.
Computing the evidence can be computationally intensive and usually requires approximation.
What is more, the Bayes factor is very sensitive to the choice of prior and can result in estimates that heavily underﬁt the data (especially under a vague prior, as shown in Appendix E), having the tendency to select the simpler model; which is argued further by M. S. Bartlett (1957) and Akaike et al.
This is handled in Ghahramani & Heller (2006) by using the empirical Bayes procedure, a shortcoming of which is the issue of double counting (Berger, 2000) and thus being prone to over-ﬁtting. We address these issues by choosing to work with information criteria based model comparison as opposed to using the Bayes factor. The details are described in Section 3.
We are not aware of any prior work on sentence similarity that uses our approach.
We employ a generative model for sentences similar to Arora et al.
, but our contribution differs from theirs in that our similarity is based on the aforementioned model comparison test, whilst theirs is based on the inner product of sentence embeddings derived as maximum likelihood estimators.
The method by Marshall et al.
(2006) applies the same score as in Equation 1 in the context of merging datasets whilst we focus on information retrieval and semantic textual similarity.
M ETHODOLOGY
We address the shortcomings of the Bayes Factor described in Section 2 by proposing model comparison criteria that minimise Kullback-Leibler (KL) divergence (denoted in equations as D KL ) across a candidate set of models.
This results in a penalised likelihood ratio test which gives competitive results. This approach relies on Information Theoretic Criteria to design a log-ratio-like test that is prior free and robust to overﬁtting.
We seek to compare models M 1 and M 2 using Information Criteria (IC) to assess the goodness of ﬁt of each model.
There are multiple IC used for model selection, each with different settings to which they are better suited.
The IC which we will be working with have the general form
[Equation]
where L (ˆ θ |D , M ) = (cid:80) ni L (ˆ θ | w i , M ) is the maximised value of the log likelihood function for model M , α is a scalar derived for each IC , and Ω ( D , M ) represents a model complexity penalty term which is model and IC speciﬁc.
Using this general formulation for the involved information criteria yields the following similarity score:
[Equation]
The paragraph surrounding Equation (9) of Tenenbaum & Grifﬁths (2001) (Tversky’s contrast model) describes what a good similarity is from a cognitive science perspective.
Interestingly, the semantic interpretation of the similarity we have derived is similar to the one described in that work.
We want to contrast the commonalities of the two datasets (through shared parameters) to the distinctive features of each dataset (through independently ﬁt parameters).
Examples of these criteria can be put into two broad classes.
The Bayesian Information Criterion (BIC) is an example of an IC that approximates the model evidence directly, as deﬁned in Schwarz et al.
Empirically it has been shown that the BIC is likely to underﬁt the data, especially when the number of samples is small (Dziak et al., 2012). We provide additional empirical results in
Appendix E, showing BIC is not a good ﬁt for the STS task as sentences contain a relatively small number of words (samples). Thus we focus on the second class - Information Theoretic Criteria.
3. I NFORMATION T HEORETIC C RITERIA
The Information Theoretic Criteria (ITC) are a family of model selection criteria.
The task they address is evaluating the expected quality of an estimated model speciﬁed by L (ˆ θ | w ) when it is used to generate unseen data from the true distribution G ( w ) , as deﬁned in Konishi & Kitagawa (2008b).
This family of criteria perform this evaluation by using the KL divergence between the true model G ( w ) and the ﬁtted model L (ˆ θ | w ) , with the aim of selecting the model (from a given set of models) that minimizes the quantity
[Equation]
The entropy of the true model H G ( w ) will remain constant across different likelihoods.
Thus, the quantity of interest in the deﬁnition of the information criterion under consideration is given by the expected log likelihood under the true model E G [ln p ( w | ˆ θ )] .
The goal then, is to ﬁnd a good estimator for this quantity.
One such estimator is given by the normalized maximum likelihood:
[Equation]
where ˆ G represents the empirical distribution.
This estimator introduces a bias that varies with respect to the dimension of the model’s parameter vector θ and requires a correction in order to carry out a fair comparison of information criteria between models.
A model speciﬁc correction is derived resulting in the following IC, called Takeuchi Information Criterion (TIC) in Takeuchi (1976)
[Equation]
Then, under the TIC we have the following similarity (full derivation in Appendix A)
[Equation]
For the case where we assume our model has the same parametric form as the true model and as n → ∞ , the equality ˆ I =
ˆ J holds resulting in a penalty of tr (cid:16)
ˆ I ˆ J − 1 (cid:17) =
tr( I k ) =
k where k is the number of model parameters, as shown in Konishi & Kitagawa (2008a).
This results in the Akaike
Information Criterion (AIC) [Equation] Akaike, 1974)
[Equation]
The AIC simpliﬁcation of TIC relies on several assumptions that only hold true in the big data limit. AIC also assumes our model M has the same parametric form as the true model (Konishi & Kitagawa (2008a)).
In general, TIC is a more robust approximation.
However, as shown in
Appendix F, for models with a high number of parameters, TIC may prove unstable and thus AIC will generally perform better.
In this study we will consider and contrast both.
L IKELIHOODS
In this section we will illustrate how to derive a similarity score under our ITC framework by choosing a likelihood function that incorporates our prior assumptions about the generating process of the data.
Adopting the viewpoint of a practitioner, we would like to compare the performance of two models — one that ignores word embedding magnitude, and one that makes use of it.
Our modelling choices for each assumption are the von Mises-Fisher (vMF) and Gaussian likelihoods respectively.
The comparison between the two likelihoods we provide in Section 5 provides empirical evidence as to which approach is better suited to modelling word embeddings.
The rest of this section outlines the details of each of the two modelling choices.
4. V ON M ISES -F ISHER L IKELIHOOD
Word embeddings are d dimensional vectors of real numbers, that are traditionally learned by optimizing a dot product between target words and context vectors (Mikolov et al., 2013).
Due to this training setup, cosine similarity is often used to measure the semantic similarity of words in various information retrieval tasks.
Thus, we want to explore a distribution induced by the cosine similarity measure.
We model our embeddings as vectors lying on the surface of the d − 1 dimensional unit hyper-sphere w ∈ S
d − 1 and i.i.d. according to a vMF likelihood (Fisher et al.
[Equation]
where µ is the mean direction vector and κ is the concentration, with supports || µ
||
=
||
w ||
= 1 , κ ≥ 0 .
The term I ν ( κ ) Corresponds to a modiﬁed Bessel function of the ﬁrst kind with order ν .
In this work we reparameterise the random variable to polar hypersphericals w ( φ ) ( φ = ( φ 1 , ..., φ d − 1 ) (cid:62) ) as adopted in Mabdia (1975).
Further details can be found in Appendix B.
The ﬁrst and second order partial derivatives of the vMF log likelihood are derived in Appendix C.
We prove (in Appendix C) that the mixed derivatives of the vMF log likelihood are a constant (with respect to φ ) times ∂ L ( θ , κ | φ ) /∂θ k .
Thus, evaluated at the MLE, these entries are zero. Assuming l < k
[Equation]
Thus, ˆ J is a diagonal matrix, with diagonal j = ( ˆ J 11 , ..., ˆ J
[Equation]
(cid:62)
[Equation]
This quantity only requires O ( nd ) operations to compute and thus does not increase the asymptotic complexity of the algorithm.
The closed form of the similarity measure for two sentences D 1 , D 2 of length m and l respectively under this model is then
[Equation]
where the Jacobian terms (from the reparametrisation) cancel out. The subscripts indicate the sentence, with 1 , 2 meaning the concatenation of the two sentences. 4. G AUSSIAN LIKELIHOOD
Schakel & Wilson (2015) show that some frequency information is contained in the magnitude of word embeddings. This motivates a choice of a likelihood function that is not constrained to the unit hyper-sphere and possibly the simplest such choice is the Gaussian likelihood. Due to the small size of sentences 2 we choose a diagonal covariance Gaussian models. The Gaussian likelihood is then
[Equation]
where Σ is a diagonal matrix. As with the vMF likelihood, we prove in Appendix D that the Hessian of the log likelihood evaluated at the MLE is diagonal. The TIC correction is a sum of O ( d ) terms, similar to the form in Equation 2. This can be nicely written in terms of biased sample kurtosis (denoted ˆ κ )
[Equation]
The closed form of the similarity measure for two sentences D 1 , D 2 of length m and l respectively under this model is then
[Equation]
where the subscripts indicate the sentence, with 1 , 2 meaning the concatenation of the two sentences.
E XPERIMENTS
We assess our methods’ performance on the Semantic Textual Similarity (STS) datasets 3 (Agirre et al., 2012; 2013; 2014; 2015; 2016).
The objective of these tasks is to predict the similarity between two given sentences, validated against human scores.
In our experiments, we assess on the pre-trained GloVe (Pennington et al., 2014), FastText (Bojanowski et al., 2016), and Word2Vec
(Mikolov et al., 2013) word embeddings.
For the vMF distribution, we centre the distribution of word embeddings and normalize them to be of length 1.
Some of the sentences are left with a single word after querying the word embeddings, making the MLE of the κ parameter of the vMF and Σ parameter of the Gaussian ill-deﬁned.
We overcome this issue by padding each sentence with an arbitrary embedding of a word or punctuation symbol (i.e. ’.’
or ’?’).
We ﬁrst compare our methods: vMF likelihood with TIC correction (vMF+TIC) and diagonal Gaussian likelihood with AIC correction (Diag+AIC) against each other. Then, the better method is compared against mean word vector (MWV), word mover distance (WMD) (Kusner et al., 2015) 4 , smooth inverse frequency (SIF), and SIF with principal component removal as deﬁned in Arora et al.
We re-ran these models under our experimental setup, to ensure a fair comparison.
The metric used is the average Spearman correlation score over each dataset, weighted by the number of sentences.
The choice of Spearman correlation is given by its non-parametric nature (assumes no distribution over the scores), as well as measuring any monotonic relationship between the two compared quantities. Embedding Method [Table]
Average FastText vMF+
[Table] [Table] [Table]
Diag+ [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
GloVe vMF+TIC 0.
[Table]
Diag+AIC 0.
Word2Vec
vMF+TIC 0.
Diag+AIC 0.
[Table]
[Table]
[Table]
[Table]
average is weighted according to dataset size.
5. E MBEDDING MAGNITUDE
We present the comparison between the similarities derived from a vMF and Gaussian likelihoods in Table 5. We note that the Gaussian is a much better modelling choice, beating the vMF on every dataset by a margin of at least 0.05 (5%) on average, with each of the three word embeddings. This is strong evidence that the information encoded in an embedding’s magnitude is useful for tasks such as semantic similarity. This further motivates the conjecture that frequency information is contained in word embedding magnitude, as explored in Schakel & Wilson (2015).
5. O NLINE SCENARIO
In Section 1, it was mentioned that in an online setting, one cannot perform the principal component removal described in Arora et al.
(2016), as that pre-processing requires access to the entire query dataset a priori.
This scenario arises in the context of information retrieval, e.g. when creating a chat-bot.
The comparison against the baseline methods are presented in Table 5.2.
We are able to out-perform the standard weighting induced by mean word vector, as well being on par with the smooth inverse frequency weighting technique (SIF). We outperform SIF using the GloVe representations by 0.0278 ( 2 . 78% ), effectively tie in the FastText representations with a difference of 0.0021 ( 0 . 21% ) and we are marginally below using Word2Vec by 0.0028 ( 0 . 28% ).
Overall the only signiﬁcant difference is 2 .
78% under GloVe word vectors where we outperform SIF.
5. O FFLINE SCENARIO
Word vectors Method STS
STS
STS
STS STS Average FastText
Diag+AIC 0.
SIF+PCA 0.
GloVe
Diag+AIC 0.
SIF+PCA 0.
Word2Vec
Diag+AIC 0.
SIF+PCA 0.
There are use-cases in which the entire dataset of sentences is available at evaluation time — for example, in clustering applications.
For this scenario, we compare against the SIF weightings, augmented with the additional pre-processing technique seen in Arora et al. We need only consider this baseline, as it outperforms all others by a large margin.
The results are shown in Table 5.3.
We remaing competitive with SIF+PCA on all three word embeddings, being able to match very closely on GloVe embeddings. On the FastText and Word2Vec embeddings, our method is less than 0.01 (1%) lower than SIF+PCA.
D ISCUSSION
We’ve presented a new approach to similarity measurement that achieves competitive performance to standard methods in both online and ofﬂine settings.
Our method requires a set of clear choices — model, likelihood and information criterion.
From that, a comparison framework is naturally derived, which supplies us with a statistically justiﬁed similarity measure (by utilizing ITC to reduce the resulting model-comparison bias).
This framework is suitable for a variety of modelling scenarios, due to the modelling freedom in specifying the generative process.
The graphical model we employ is adaptable to encode structural dependencies beyond the i.i.d.
data-generating process we have assumed throughout this study — for example, an auto-regressive (sequence modelling) model may be assumed if the practitioner suspects that word order matters (i.e. compare ”Does she want to get pregnant?” to ”She does want to get pregnant.”) In this study, we found that the von Mises-Fisher distribution lends itself to representing word embeddings well, if their magnitude is disregarded and a unimodal distribution over individual sentences is assumed.
Relaxing the former assumption, we can model word embeddings with a Gaussian likelihood. As this improves results, this is empirical evidence that word embedding magnitude carries relevant information, which agrees with prior intuition built from Schakel & Wilson (2015). We hope that this framework could be a stepping stone in using more complex and accurate generative models of text to assess semantic similarity. For example, relaxing the assumption of unimodality is an interesting area for future research
The MLE for p ( φ 1 , 2 | θ , M 2 ) can be derived by simply estimating the separate MLE solutions for p ( φ 1 | θ 1 , M 2 ) and p ( φ 2 | θ 2 , M 2 ) .
What is not as obvious is that the penalty term follows the estimation pattern.
Gradient vectors for M 2 are given by (where ⊕ is concatenation)
[Equation]
and Hessian results in a block diagonal matrix
[Equation]
with inverse
[Equation]
Computing tr( ˆ I ˆ J − 1 ) then yields [Equation]
B R EPARAMETRISATION OF THE V MF DISTRIBUTION
We reparametrise the random variable to polar hypersphericals w ( φ ) ( φ = ( φ 1 , ..., φ d − 1 ) (cid:62) ) as adopted in Mabdia (1975)
[Equation]
where
[Equation]
This reparametrisation simpliﬁes the calculation of partial derivatives.
The maxima of the likelihood remains unchanged since | ∂ w /∂ φ | does not depend on θ thus the MLE estimate in the hypershperical coordinates parametrisation is given by applying the map from the cartesian MLE to the polars.
[Equation]
Let D =
{ φ i } ni =1 be the dataset.
The log likelihood is then
[Equation]
C P ARTIAL D ERIVATIVE C ALULATIONS ( V MF LIKELIHOOD )
We ﬁrst show the following result, which is useful for the full derivation.
For k ≤ j
[Equation]
where the second line comes from the fact that sin θ k (or cos θ k ) gets transformed into a cos θ k (or − sin θ k ) [Equation] and thus we can revert to the original deﬁnition of µ j by multiplying with a cot θ k (or − tan θ k ).
If k > j , this derivative is 0.
Thus, for a single data point w ( φ )
[Equation]
where the sum starts from k , as for j < k , the derivative is zero.
The derivative with respect to κ is derived as follows
[Equation]
where the derivative of the second term is a known result. We next focus on second order derivatives
[Equation]
Unless this derivative is zero, we notice that we take the derivative ∂ 2 cos θ k ∂θ
[Equation]
or ∂
[Equation]
sin
[Equation]
k ∂θ 2 k , both of which result in the negative of the original function.
Thus
[Equation]
The below result is given
[Equation]
Next, we show that the second order mixed derivatives are a constant (with respect to φ ) times ∂ L ( θ , κ | φ ) /∂θ
k , i.e.
[Equation]
Evaluated at the MLE by deﬁnition
[Equation]
Assuming l < k (Hessian is symmetric)
[Equation]
Which is also zero evaluated at the MLE cot θ l ∂ L ( θ ,κ |D )
∂θ k
[Equation]
D P ARTIAL D ERIVATIVES C ALCULATION (G AUSSIAN L IKELIHOOD )
The partial derivatives for the diagonal Gaussian likelihood are (we take derivatives with respect to precision λ 2 k = 1 /σ 2 k )
[Equation]
Evaluating at the MLE we get
[Equation]
Substituting these derivatives into the deﬁnition of ˆ I and ˆ J we get
[Equation]
Finally
[Equation]
computing the model complexity penalty we get
[Equation]
E B AYES F ACTOR AND B AYESIAN I NFORMATION C RITERION
We ﬁrst deﬁne the Bayes Factor for the Gaussian likelihood with parameters ( µ , Λ = Σ − 1 ) .
We assume a Wishart prior
[Equation]
which yields the following Normal-Wishart posterior (Murphy, 2007)
[Equation]
The posterior parameters are
[Equation]
The evidence (Murphy, 2007) is
[Equation]
Using p ( D 1 , D 2 |M 1 ) =
p ( D 1 ⊕ D 2 |M 1 ) we compute the Bayes factor for M 1 , M 2 in closed form
[Equation]
) where |D 1 | = n, |S 2 | = m and |D 1 ⊕ D 2 | = m + n = l .
The BIC is deﬁned as
[Equation]
and acts as a direct approximation to the model evidence (Schwarz et al., 1978).
Thus, the similarity under the BIC is
[Equation]
where n, m are deﬁned as above.
Equations 3 and 4 represents our similarity score under a Gaussian likelihood, for the Bayes Factor and BIC respectively.
Table E compares BIC and the Bayes Factor using a Gaussian likelihood to the approach presented in the paper.
We see that while these approaches are competitive on STS-13, STS-14 and STS-15, they both give severely worse results on STS-12 and STS-16, with more than 0.08 difference.
This motivates our choice to do a penalised likelihood ratio test instead of doing full Bayesian inference of the evidences.
[Table]
F TIC R OBUSTNESS
In this section, we compare the TIC and AIC on the two likelihoods described in the main text.
Table
F presents the results of that comparison.
As we can see, with each word embedding the Gaussian
AIC correction outperforms the TIC correction on average.
Looking at Figure 3, it becomes apparent why — the more parameter a Gaussian has, the more dependent its correction is on the number of words in the sentence.
This is reminiscent of the BIC penalty discussed in Appendix E, which was shown to perform badly.
On the other hand, looking at Figure 4, we see that the TIC for the vMF distribution has very low variance, and is generally not dependent on the number of word embeddings in the word group.
This gives intuition why the AIC and TIC for the vMF give very similar results.
Embedding
[Table]
between the TIC and AIC corrections.
[Figure]
[Figure]